Judgement vs. testing

John Gruber points to a short piece by Aaron Swartz who calls my earlier post Buzz launch wasn’t flawed, Google’s intentions are “dumb.” Here’s the passage he quotes:

Google is a $170 billion company. It employs thousands of engineers and developers. It tests, tests, tests, and tests more. In fact, its “designers” once unable to pick a shade of blue tested 41 variations of it. It’s ludicrous to think that the Buzz fiasco was simply a result of under-testing.

Swartz dismisses it with:

Yes, Google tests lots of minor details with lots of user data. How do they get this data? From actual users. How do they get actual users? By releasing products. So it seems totally reasonable to imagine them releasing something without heavily testing it; their whole culture is based around testing things in the wild.

I’m really not sure what exactly Swartz finds “dumb,” my claim or Google’s “whole culture” of “testing things in the wild”? Either way, I smell a “Rookie Designer 101” engagement here.

First, some facts regarding Google:

  1. Google runs the third largest email operation on the planet.
  2. Google holds perhaps more personal data of various types than any other NGO in the world.
  3. Privacy of personal data has been a central issue for Google for the past several years, notably in the U.S., Europe and China.
  4. Google has repeatedly entered and failed the social networking arena.
  5. Google Wave, whose enduser functionalities overlap quite a bit with Google Buzz, has not been warmly received.
  6. Google is a $170 billion company with absolutely no shortage of resources (except strategic designers).
  7. Google did admit failure and backtracked on Buzz.

So, don’t listen to me. Google itself accepts there was failure. But of what? That remains the question.

First launch, then test and then call it beta?

According to Swartz it’s “totally reasonable” to release products and then test on actual customers. You would do that if you had no taste, no judgement and no experience in design and your customers accepted to be guinea pigs for you.

Exposing what was up to that point users’ private email network by default and then, faced with an angry userbase, suggesting convoluted series of instructions to remedy it is indicative of two equally sad possibilities:

  1. Google doesn’t have sufficient corporate leadership with taste, judgement and experience to stand up to the expedient Buzz product guys and say, “We’re the ‘Don’t be evil’ company, we don’t violate 175 million users’ trust in us for privacy for anything, period.” (If Google needs lessons on what happens when security/privacy becomes an afterthought in product design, it should look no further than the company it’s begun to methodically emulate in Redmond.)
  2. That the CEO of Google hasn’t yet apologized for the incident is frightening given Google’s ambitions, considering facts #1, #2 and #3 above. (Even the reclusive Steve Jobs personally apologized for MobileMe’s operational failure, gave rebates and made sure it was back on track.)

I don’t know which one is more disturbing, that every single one of the thousands of people at Google who tested Buzz internally lacked the taste, judgement and experience necessary to know privacy/security should never be opt-out or that if they did, management overrode their judgement for the sake of expediency to piggyback naked on Gmail by default.

You can’t test everything. Neither should you.

If every aspect of design was testable, there’d be no reason to have designers, pattern algorithms would automate it completely. (Perhaps that’s what Google would like.)

When designers do test, they do so because they themselves are not sure what works best in a given context. Testing informs designers, it doesn’t magically produce judgement. If Google needs to be taught in 2010 by testing on actual users without prior consent that the core tenet of a private/secure system design must not require opt-out, then shame on them.

That’s Not How or What You Test, Mr. Swartz, that’s how you expose your greed or incompetence. You be the judge.

18 thoughts on “Judgement vs. testing

  1. @Alain – brilliant!

    @Berend – I agree yet Kontra’s point still stand and Frankenstein’s lab?! …..priceless!

  2. No, google is not the company of scientists, it’s the company of Frat boys. Intelligent frat boys, but frat boys nonetheless.

  3. What is Google?

    By conception Google is the ultimate serendipity company. By coincident two students happened to discover an incredibly strong search algorithm in a PhD project. When they made it a start-up they, just by chance, connected it to a winning business formula.
    No plan. No vision. Not even serious business ambitions. Just smart guys, having fun in doing science.
    When you look at whatever happened since, it seems to be just an extension to this first beginning. When Googleplex looks like anything, it is a big university campus where smart guys are doing fun projects with fun experiments.
    Apple is the company of a visionair. Microsoft is the company of a businessman. Google is the company of scientists. Wherever that might lead us…

  4. Aaron
    When your second statement does not follow your first, one is apt to entertain qualms about the completeness of your understanding of the problem: “testing things in the wild” is contra-indicated.


    • Speaking of “reading comprehension problems”, Buzz has been a “huge disaster” not because it didn’t have green backgrounds or red buttons (the category of issues that can be surfaced by external testing and corrected), but because of one fundamental failure: default opt-out policy.

      Design, interaction, and even workflow co-mingling problems I referred to can be corrected by testing and re-work, but “opt-out by default” is another animal altogether. It needs no testing whatsoever.

      Let’s repeat that. In 2010, the concept needs no testing by the 3rd largest email company and the biggest information NGO in the world. They should have known and I’m sure they do know. They simply tried to cheat their way to the top of the social networking business on top of Gmail, at the expense of Facebook and Twitter. It’s as simple as that. It exposes Google’s intent. It goes to the heart of the matter: trust.

      The ratio of internal vs. external testing is a red herring, which I’m sure Google would like you to get bogged down in just so that you miss their real crime here: the intent to sacrifice their customers’ security/privacy in a mad rush to beat Facebook and Twitter.

      In case you misread it, that was the main theme of my piece.

  5. In google’s mind they are no breaking your privacy, instead
    they are sharing with their ad buyers customer meta information
    just like telephone companies, grocery store, etc.

    This is no way a defense of advertisement model but that
    is how corporation brain wash America for last 60 years.

    All you has to do is watch BBC’s Centure of Self to see
    how this system was developer and used by US political system.

  6. Either way Google released the entire connections graph of gmail to spiders / crawlers bots etc. Do you think in the time it takes to opt out that a bot has already scraped it? they did it intentionally or they’re idiots with a shye load of private data and are Very Nuclear Dangerous. the thirst to me-to facebook was worth the myopic decision. I am pissed schmidt hasn’t apologized.

    • I’m really not sure what exactly Swartz finds “dumb,” my claim or Google’s “whole culture” of “testing things in the wild”?

      This piece answers the first possibility.

  7. I don’t think Swartz would disagree with much here. This seems to be a post about semantics than anything of substance. I’m sure by “reasonable” Swartz means “expected”, but you seem to take this as “justified.” Put that out of your mind.

  8. I have to say I was pretty puzzled by Mr. Swartz’s post.
    Does he really think a huge public release pushed on every user of Gmail is legitimate “testing”? Even putting aside the jokes about Google’s Beta programs, almost all of them were extremely limited/invite only for months if not years. Buzz was just pushed out to everyone. Huge difference.

  9. Opt-in betas are the way to do testing in the field and then only after extensive testing is done in house.

    I agree with you completely. Google did this on purpose to become the second largest social network overnight with complete disregard for user’s privacy.

    This being the launch of a major “product” top executives where definitely involved and either gave their blessing or pushed their weight around to have it that way.

    The more you think about this the more obvious it becomes that this was done intentionally. People who are taking Google’s side on this are either naive or fan boys/girls.

    I think this event has not only tainted Google’s reputation but all cloud services. Since Google was considered by some as the epitome of don’t be evil and they did this, how can users trust other smaller less known SaaS providers.

  10. What a typically insightful and right-on post.

    The only thing I hate about this blog is not having found it until just this week – so many old posts to read. ;-)

  11. Regarding Swartz’s comment: “From actual users. How do they get actual users? By releasing products. So it seems totally reasonable to imagine them releasing something without heavily testing it; their whole culture is based around testing things in the wild.”

    I would say off the top of my head that when Google releases products to “test” in the wild they do so as beta. Chrome browser is still beta. Gmail was the longest beta I can remember. Wave is in “limited preview” which sounds like a beta release to me. When YouTube offered HTML5 to replace flash they did it as an opt in. They could easily have done the same with Buzz and didn’t. So I don’t find Swartz’s point compelling. IMO they wanted a product out there to plant the Google flag on the social media turf and testing was not at all the point of the release.

Comments are closed.