February 2nd, 2009
10:17 AM ET

What Google did right

[cnn-photo-caption image=http://i2.cdn.turner.com/cnn/2009/TECH/01/28/google.offline.gmail/art.Google2.gi.jpg]

David Gewirtz | BIO
Editor-in-Chief, ZATZ Publishing

On Saturday, Google killed the Internet - at least for about an hour. Saturday morning, between 9:30a.m. and 10:25a.m. Atlanta time, no matter where you were in the world, if you did a Google search, you'd see the message, "This site may harm your computer" returned for each result.

Given that more than half of all Internet searches are done using Google, this effectively meant that most Web accesses resulted in a dire warning. That was obviously a serious error in Google's operation. What's most interesting, though, is how well Google handled the problem.

As it turns out, a very simple technical glitch caused the error. Google gets a list of malware sites (sites that actually will harm your computer if you visit them) from a non-profit organization operated by two Harvard professors. This list is updated constantly, and a data file containing that list is regularly transmitted to Google.

This time, instead of just a list of bad site URLs, the file contained one additional item, a slash. That's right, the simple / symbol. That one character, to a computer network, literally means "everything". When the file was imported, it read the slash, and dutifully proceeded to label everything, every single site on the planet, as a bad site.


Let's be clear. There are going to be problems like this in any complex system and Google's is a system of almost incomprehensible complexity. This was a dumb programming mistake that can be fixed with only one or two lines of code, but that's the case with almost all programming mistakes. Google is now well-aware of this particular bug and you can bet it's already fixed.

Here's what Google did right

First, Google fixed the problem almost instantly. The problem began to manifest at 6:30a.m. on the West coast, where Google is headquartered. At 6:30a.m., most good programmers are either sound asleep - or still up from the night before. Waking up at 6:30a.m. is hard.

And yet, within 55 minutes, starting at 6:30a.m. on a Saturday, Google not only discovered there was a problem, they marshaled their troops, identified what was happening, and implemented a fix.

The next thing Google did right was own up to the problem. But even better, they provided a detailed explanation of what went wrong. If they'd been like most companies, they'd have said they found a problem and fixed it, with no explanation.

When faced with a problem, most companies (and, uh, governments) respond without detail. This always leaves lingering doubt. What caused it? Is this just the beginning? Are they falling apart? And so forth.

But by providing a detailed enough explanation that technical people and non-technical people could grasp the problem and understand it wasn't a fundamental problem to the Google infrastructure. It was stupid, to be sure, but not serious.

The final thing Google did right was put a real person's signature at the bottom of their mea culpa. Marissa Mayer is Google's VP of Search Products & User Experience and as one of their first employees, has a net worth of almost staggering proportions. And yet, she signed her name to Google's official statement explaining the problem.

By signing such a statement with an executive's name, this shows that Google management understands what's happening and also shows it takes responsibility for the problem.

These three things - fixing the problem very quickly, explaining it in sufficient detail, and showing a senior executive "owns" the explanation - helps us understand that while this was a dumb mistake, it's one that's under control.

There's no doubt this lil' problem will be a PR issue for Google for the next week or so. But because they handled it in such an excellent way, it will blow over, merely to become another war story told at cocktail parties. There will be no long-term damage to either Google or the public's confidence in Google's systems.

In fact, because Google responded so well, this incident might even increase the public's confidence in Google's ability to respond to technical problems with both technical skill and forthrightness.

Google's behavior in this incident should be a role model for other companies - and, dare I say, our government as well.

Editor’s note: David Gewirtz is Editor-in-Chief, ZATZ Magazines, including OutlookPower Magazine. He is a leading Presidential scholar specializing in White House email. He is a member of FBI InfraGard, the Cyberterrorism Advisor for the International Association for Counterterrorism & Security Professionals, a columnist for The Journal of Counterterrorism and Homeland Security, and has been a guest commentator for the Nieman Watchdog of the Nieman Foundation for Journalism at Harvard University. He is a faculty member at the University of California, Berkeley extension, a recipient of the Sigma Xi Research Award in Engineering and was a candidate for the 2008 Pulitzer Prize in Letters.

Filed under: David Gewirtz • Technology
soundoff (9 Responses)
  1. melbro

    This is pretty amazing that a slash can create problems such as these. I didn't even hear about this until now.

    February 2, 2009 at 6:44 pm |
  2. Pati Mc

    If we are fortunate, someone from the current adminsitration will sit down with the good folks at Google and gain some wisdom. (This is not to imply that they are not doing a good job, but there are valuable lessons to be learned in how this event was handled that could prove useful across the board and across the aisle.)

    February 2, 2009 at 5:56 pm |
  3. Monica from VA

    This is very interesting information and the first time I am hearing about it. Interesting enough, since Saturday, my browser (Firefox) was redirecting me to a different site every time a search result was clicked. It turned out to be the "Zlob" virus (affects only firefox browsers, I believe). I'm sure this is just a coincidence, but if anyone else has this problem too, that is the name of the virus. I used combofix to finally remove it.

    February 2, 2009 at 5:25 pm |
  4. Isabel Abreu

    I was browsing, quietly, in the web, in the sunny morning of Saturday (38 degrees and very air-conditioning for my Dell doesn't stop), when encountered me with such a disaster.
    All my sites come with the message "This site may harm your computer" in Google searches. I get in IE, merge to Google Chrome, and nothing!

    After almost stroke, I see that all is well. So Google and UOL may fail.
    Well, this failure turned comedy. I wasn't alone.

    This failure, Google's is fun for people to geek like me, when I understand that it isn't personal with my work, but how it must have been the majority of users 'home'? Some people should just stop surfing and get traumatized for a long time ...

    Who works in the area of systems, and never made a grotesque mistake, still has much to learn. This is part.

    To err is human!
    Google, for me, you are forgiven!

    February 2, 2009 at 2:08 pm |
  5. Steve in Las Vegas,NV

    Great news! They explained the problems in ENGLISH, instead of Microsoft's usual compubable(understood only by the programmers who wrote the program) and without any explaining what happened.

    We all know there is going to be problems from time to time, but Google jumped on it promptly, got it fixed and let everybody know what happened...How refreshing! If only others did this as well!

    February 2, 2009 at 12:44 pm |
  6. Annie Kate

    Google isn't the only computing company who responds like this. The computing department I worked in handled production support problems all the time – we had a good turnaround time (hours not days – sometimes mere minutes) and we always had a detailed report out within a couple of hours to our manager who signed off on it which detailed root cause, contributing factors, and what we did to fix it and what impact it had. This way of doing things is just good business practice and should be the rule and not the exception.

    February 2, 2009 at 12:13 pm |
  7. Mary V., Salt Lake City, UT

    IT IS FUNNY....... that the "CONservative" site thought it was a Liberal attack!

    The CONservatives, are now believing their OWN LIES!!!

    February 2, 2009 at 12:13 pm |
  8. James in Nebraska

    I find it hilarious how on one conservative site they thought it was a liberal attack. They claimed Google and their liberal agenda were out for them. They kind of jumped the gun and didn’t realize all websites were tagged this way I guess.

    February 2, 2009 at 11:34 am |
  9. Michael "C" Lorton, Virginia

    And that is why they are called "Google"

    February 2, 2009 at 11:22 am |