OUCH! Microsoft fails to back-up, loses customer T-Mobile data

[berni_mccoy]

http://www.techcrunch.com/2009/10/10/t-mobile-sidekick-disaster-microsofts-servers-crashed-and-they-dont-have-a-backup/
T-Mobile Sidekick Disaster: Danger’s Servers Crashed, And They Don’t Have A Backup

Wow. T-Mobile and Danger, the Microsoft-owned subsidiary that makes the Sidekick, has just announced that they’ve likely lost all user data that was being stored on Microsoft’s servers due to a server failure. That means that any contacts, photos, calendars, or to-do lists that haven’t been locally backed up are gone. Apparently if you don’t turn off your Sidekick and make sure its battery doesn’t run out you can salvage what’s currently stored on the device, otherwise you’re out of luck: Microsoft/Danger is describing the likelihood of recovering the data from their servers as “extremely low”.

T-Mobile Sidekick users have been sufferingfrom a major outage all week, and that issue apparently hasn’t been resolved either.

This goes beyond FAIL, face-palm, or any of the other internet memes we’ve come to associate with incompetence. The fact that T-Mobile and/or Microsoft Danger don’t have a redundant backup is simply inexcusable, especially given the fact that the Sidekick is totally reliant on the cloud because it doesn’t store its data locally. Microsoft acquired Danger for $500 million in February 2008.

I'll still give this a face-palm:

[hobbit709]

1. Microsoft wants you to believe their OS never fails

It's always the customer's or the hardware's fault. Evidently they believed their own BS.

I trust no external source with my data.

[still_one]

6. Microsoft has never said that, in fact EVERY software manufacter excludes loss of data

As far as trusting no external source with your data, you are absolutely correct, and that is why I do my own backups on my OWN storage for both my Microsoft and Apple

[Statistical]

10. Please link to a single cite where Microsoft has ever said that.

Many portions of the MCSE series of exams deal with backups, redundancy, restoration, and catastrophic failure. If anything via the certification criteria Microsoft is making it well known both hardware and software fails (as well as human failure)

[hobbit709]

15. Ever deal with MS Help

The OS is never at fault-you did something wrong or the hardware is at fault.

They're like the one place where I worked about10 years ago. The inventory tracking system had a display on the bottom of the screen defining the various function keys. My fave "For help press F1"
When you press F1 a screen pops up that says "There is no help for this subject"

[Statistical]

16. Did it ever occur to you that the inventory tracking system wasn't made by Microsoft?

Sounds like a shitty programmer working for whoever made that inventory tracking software?

It is Microsoft's fault how?

[hobbit709]

24. Analogy, not literalism.

[Turbineguy]

2. I've suggested to Microsoft

that when people call for technical help their call is answered in less time than it takes to install Linux

[cliffordu]

3. Perfect:

Microsoft is gift that just keeps on giving.

I love stories like this - makes my schadenfreude all warm and runny.

[SpookyCat]

4. Facepalm indeed!

That's just pathetic. Did they overlook it, or was this arrogance? Either one I could believe.

I drill this into my clients, backup backup backup...

Well, as long as M$ is incompetent my job is safe. Sooooo...my job is safe.

[SheilaT]

5. This is better thought of as

why storing all data off-site is a bad idea.

Some ten or fifteen years ago I knew a guy who was wildly enthusiastic about computers that would have no hard drive at all, but tha all work would be done and stored off-line. I thought it was a terrible idea, that the possibility of security breaches or lost work was practically infinite. And here we are.

[MyNameGoesHere]

7. Fucking rubbish.

Servers fail, and some dimwit IT department didn't back up. And this is MS fault how? Even worse the data is probably stored on a SAN that should be redundant, MS doesn't make SAN's nor do they make Enterprise Disaster recovery/back-up software. More small minded carping on MS about nothing.

[Statistical]

9. The issue is microsoft OWNS Danger.

They bought them 2 years ago.

How/why someone paid by Microsoft and likely making $300K - $500K a year could be so stupid as to not backup the SAN before doing a SAN-wide upgrade is simply mind boggling.

While no Microsoft servers failed as the OP seems to indicate there was a failure at Microsoft... a human failure.

[sudopod]

11. If they didn't make backups for a 500 million dollar subsidary

then you can bet your ass someone high up signed off on the lax backup policy.

[berni_mccoy]

12. Corporate policies are suppose to protect against human failure, which this was

It was no doubt human failure, but procedures and policies are suppose to protect against this. It is not a simple failure of one person, but that person's team and management. And while I agree with your sentiment, it is Microsoft's responsibility.

[Grinchie]

21. Trouble is, the dimwit IT Department is likely in Bangalore.

And most likely, it certainly is MSFT's fault.

The company has been foundering for many years, especially when it started thinking that they no longer needed to support the innovators, beta testers, and solution providers and start turning them into just another consumer base.

They actually thought that the Nerds would be happy to pay MSFT for access to mountains of Knowledge Base articles in search of the elusive cause of the BSOD.

Then the vultures descended, creating the MSFT Certified Professional/No Nerd Left Behind industry, where rote learning and rigid testing were supposed to be the answer for businesses seeking qualified technicians. All it did was enable shallow thinkers into the profession, and further reduce the overall quality of support.

So long MSFT. It was a good run for 10 years, but just like any bubble, your membrane has popped, and you have embarked on the treacherous waters of IT with a ship that has no propulsion, bilge pumps or life preservers, and the current port is selling these things at 10 times the price.

[Statistical]

22. lol. MSFT is doing fine and will contiune to do fine.

Were you aware IIS marketshare has grown not shrunk over the last 10 years.

ASP.NET is now depolyed on 46% of Fortune 1000 company websites.

Hell there are more pirated copies of Windows than there are copies of Linux installed on desktops.

You are neither first nor unique in your prediction of the end of Microsoft. See you in another 10 years.

[Grinchie]

25. I actually like DotNet

As for innovation, all I see is MSFT enabling more adverts, has completely failed in regards to Trustworthy computing initiative, and still can't pull it's head out of it's ass.

There were lots of cheerleaders like you for General Motors too.

MS will never be able to win me back, despite the fact that the idiots in the fortune 1000 don't really have a choice, do they?

[Statistical]

28. I never cheerleaded GM. If you check my history I was saying GM was going into BK...

no matter what the govt did.

To compare Microsoft to GM is foolish though.

Microsoft remains and will remain a force because of developers. The amount of tools, resources, and knowledge that Microsoft providers for developers is staggering. .NET has only accelerated this trend. Business love the idea of managed code because it can be controlled. They aren't using it because "they have no choice". They are using it because it is the best platform to develop enterprise applications in a scalable, safe, and cost effective manner.

Your statement that you like dotnet but Microsoft can never win you back and calling everyone in Fortune 1000 companies an idiot are very telling.

[Statistical]

8. The issue is a little more complex and at the same time even harder to understand.

Danger like any large enterprise doesn't use a single server. Data is stored redundantly on a SAN (storage area network). The SAN isn't backed up because the SAN itself is a backup. Data is stored redundantly in multiple location, with parity recovery.

Danger decided to upgrade the SAN and hired Hitachi to do it. At that point should have known a failure in the SAN could be network wide and cause all data on all servers to be unrecoverable at the same time. For whatever reason (hard to imagine someone would ok this) they decided not to make a backup. The SAN upgrade failed and all data is unrecoverable....

How/why someone (likely someone making high 6 figures) would sign off on a network wide upgrade without either a) recognizing the risk or b)taking appropriate action to mitigate the risk.

Couple points are wrong in the article. The data wasn't "stored on Microsoft servers". Nobody does that for this much data even if Microsoft Servers are the "front end". The data was stored on a SAN (and Microsoft has no SAN products). No microsoft server failed. The nice thing about a SAN is it makes you immune to server failure. The server is simply a "data engine" and if one fails it can be replaced with another one. Normally they are already N+1 redundant so no loss of operation is possible.


Anyways it wasn't a hardware or software failure at Microsoft it was a human failure. They might as well write off the 1/2 billion investment in Danger. My understanding is they were already losing subscribers, this will annihilate the company.

[petronius]

14. I'm sitting here trying to imagine how the technicians felt at the moment they

realized that the upgrade had failed and everything was gone - I just can't do it. I know the sinking feeling when this happens at the tiny scale - taking a 'shortcut' and costing myself a couple extra hours of work - but this must have been in the cardiac arrest category.

There was probably conversation somewhere before this started:
Tech: "Do you want to make a backup?"
Manager: "How much will it cost?"
T: "Around $10,000" (I have no clue, really.)
M "No"
T: "Are you sure? We could lose everything..."
M: "I'm sure."
T: "Really sure?"
M: "Get to work, peon!"
M: "Booya! Saved $10K! Time for a martini..."

:)

[Statistical]

18. Yeah likely something like that...

Suddenly that awful horror descends where your brain realizes there is no "undo" on what you just did. :)

In hindsight the brain is instantly clear and everything (like how much of a massive risk you just took) makes sense.

[BlooInBloo]

19. If it's a live transactional server, then it's ipso facto not "the backup".

[Statistical]

23. It wasn't a server (transactional or otherwise) that failed. Try to keep up.

[hunter]

13. There's a pointy haired boss somewhere who doesn't give a shit...

... and someone above him who was patiently waiting for the project to die.

I doubt Microsoft was looking for such a spectacular and embarrassing failure, but they do buy smaller competing companies to snuff them out.

What strikes me as funny is that customers expect so little from Microsoft, and they are not outraged when systems fail. In fact most people think it's the nature of computers themselves to fail. Microsoft has trained them well. Most customers will accept "a server failure" as a valid excuse, well because, duh, in people's own experience computer systems inexplicably fail all of the time and then they have to pay money to "upgrade" them or buy new ones.

[BlooInBloo]

17. (pats his trusty google phone)

Oh well.

[Regret My New Name]

20. oops... I wonder what happened.

[Hawkeye-X]

26. Got rid of my SK3 last year..

I was getting ready to dump them in favor for Blackberries.

T-Mobile and Danger can't make ANYTHING worth shit.

Hawkeye-X

[Rex]

27. They will fix it with Windozzzz7!

Or 8 or 9...