David Gewirtz | BIO
Editor-in-Chief, ZATZ Publishing
On Saturday, Google killed the Internet - at least for about an hour. Saturday morning, between 9:30a.m. and 10:25a.m. Atlanta time, no matter where you were in the world, if you did a Google search, you'd see the message, "This site may harm your computer" returned for each result.
Given that more than half of all Internet searches are done using Google, this effectively meant that most Web accesses resulted in a dire warning. That was obviously a serious error in Google's operation. What's most interesting, though, is how well Google handled the problem.
As it turns out, a very simple technical glitch caused the error. Google gets a list of malware sites (sites that actually will harm your computer if you visit them) from a non-profit organization operated by two Harvard professors. This list is updated constantly, and a data file containing that list is regularly transmitted to Google.
This time, instead of just a list of bad site URLs, the file contained one additional item, a slash. That's right, the simple / symbol. That one character, to a computer network, literally means "everything". When the file was imported, it read the slash, and dutifully proceeded to label everything, every single site on the planet, as a bad site.
Let's be clear. There are going to be problems like this in any complex system and Google's is a system of almost incomprehensible complexity. This was a dumb programming mistake that can be fixed with only one or two lines of code, but that's the case with almost all programming mistakes. Google is now well-aware of this particular bug and you can bet it's already fixed.
Here's what Google did right
First, Google fixed the problem almost instantly. The problem began to manifest at 6:30a.m. on the West coast, where Google is headquartered. At 6:30a.m., most good programmers are either sound asleep - or still up from the night before. Waking up at 6:30a.m. is hard.
And yet, within 55 minutes, starting at 6:30a.m. on a Saturday, Google not only discovered there was a problem, they marshaled their troops, identified what was happening, and implemented a fix.
The next thing Google did right was own up to the problem. But even better, they provided a detailed explanation of what went wrong. If they'd been like most companies, they'd have said they found a problem and fixed it, with no explanation.
When faced with a problem, most companies (and, uh, governments) respond without detail. This always leaves lingering doubt. What caused it? Is this just the beginning? Are they falling apart? And so forth.
But by providing a detailed enough explanation that technical people and non-technical people could grasp the problem and understand it wasn't a fundamental problem to the Google infrastructure. It was stupid, to be sure, but not serious.
The final thing Google did right was put a real person's signature at the bottom of their mea culpa. Marissa Mayer is Google's VP of Search Products & User Experience and as one of their first employees, has a net worth of almost staggering proportions. And yet, she signed her name to Google's official statement explaining the problem.
By signing such a statement with an executive's name, this shows that Google management understands what's happening and also shows it takes responsibility for the problem.
These three things - fixing the problem very quickly, explaining it in sufficient detail, and showing a senior executive "owns" the explanation - helps us understand that while this was a dumb mistake, it's one that's under control.
There's no doubt this lil' problem will be a PR issue for Google for the next week or so. But because they handled it in such an excellent way, it will blow over, merely to become another war story told at cocktail parties. There will be no long-term damage to either Google or the public's confidence in Google's systems.
In fact, because Google responded so well, this incident might even increase the public's confidence in Google's ability to respond to technical problems with both technical skill and forthrightness.
Google's behavior in this incident should be a role model for other companies - and, dare I say, our government as well.
Editor’s note: David Gewirtz is Editor-in-Chief, ZATZ Magazines, including OutlookPower Magazine. He is a leading Presidential scholar specializing in White House email. He is a member of FBI InfraGard, the Cyberterrorism Advisor for the International Association for Counterterrorism & Security Professionals, a columnist for The Journal of Counterterrorism and Homeland Security, and has been a guest commentator for the Nieman Watchdog of the Nieman Foundation for Journalism at Harvard University. He is a faculty member at the University of California, Berkeley extension, a recipient of the Sigma Xi Research Award in Engineering and was a candidate for the 2008 Pulitzer Prize in Letters.