Google: “There has been a shift in our thinking…”

For many years when Google was under threat of regulatory action for manipulating its search results for its own commercial gain, the company used every trick in the book — including ignorance, incompetence, safe harbor, fair use, First Amendment and even web traffic beneficence — to avoid criticism in the press and investigation by regulators.

Above all, despite many examples to the contrary, Google appealed to manifest impartiality: its search results were algorithmically derived, untouched by human biases and thus fair. The list of grandiose promises and statements made by Google that turned out to be false and hypocritical is uncomfortably long. Unfortunately for the rest of us, regulatory capture being what it is and the rare penalties being laughable for a $275 billion company, there isn’t much of a black cloud left over Google to worry about, especially under the current U.S. administration.

So perhaps Google now feels freshly emboldened to tell it like it is. In any case, I was impressed by this frank admission in New York Times:

Even at Google, where algorithms and engineers reign supreme in the company’s business and culture, the human contribution to search results is increasing. Google uses human helpers in two ways. Several months ago, it began presenting summaries of information on the right side of a search page when a user typed in the name of a well-known person or place, like “Barack Obama” or “New York City.” These summaries draw from databases of knowledge like Wikipedia, the C.I.A. World Factbook and Freebase, whose parent company, Metaweb, Google acquired in 2010. These databases are edited by humans.

When Google’s algorithm detects a search term for which this distilled information is available, the search engine is trained to go fetch it rather than merely present links to Web pages.

“There has been a shift in our thinking,” said Scott Huffman, an engineering director in charge of search quality at Google. “A part of our resources are now more human curated.”

Not a shift, but a new admission of on-going reality, I’d say. Let’s hope for Scott Huffman’s sake he ran this by Google legal before it was published. Or better yet, let’s hope Google now stops the unbecoming pretensions to being philosophically open and algorithmically impartial.

Is it too hard for you?

Malarkey

When can “stuff” be called “malarkey”? Apparently, in a vice presidential debate.

We’ve always known that politics is the art of arbitrage of words and meaning. Now, researchers are using parsing methods (PDF) like the ATOS readability formula, the Lexile framework or the Flesch-Kincaid test to index “readability” against grade-levels of U.S. schools. Longer sentences and multisyllabic words indicate higher scores, shorter sentences and monosyllabic words lower scores. While there’s controversy about the validity and efficacy of these methods, they nevertheless provide some sense of textual complexity and density for historical comparisons. Here, for instance, are the Flesch-Kincaid scores of some well-known texts:

Flesch kincaid scores

Ferris Bueller’s Day Off

If we go by recent findings, most of that list remains intellectually opaque to our youth. For example, in one large-scale survey (PDF) of the top 40 books being read in schools by 9th-12th graders, the average reading level was 5.3, not much above the 5th grade. Even more damning was the fact that when librarians compiled the “Top 25 Librarians’ Picks by Interest Level” from 800 books for high school students, recommended titles were at 4th or 5th grade reading levels.

Surely, politicians do better?

One would think those whose livelihood depends on their communication skills would do better. “That’s a lot of malarkey,” as our vice president would put it. A Flesch-Kincaid analysis by Sunlight Foundation found that while lawmakers still speak above the average American (who reads at 8th-9th grade level) Congress now speaks at “almost a full grade level lower than it did just seven years ago”:

Today’s Congress speaks at about a 10.6 grade level, down from 11.5 in 2005. By comparison, the U.S. Constitution is written at a 17.8 grade level, the Federalist Papers at a 17.1 grade level, and the Declaration of Independence at a 15.1 grade level. The Gettysburg Address comes in at an 11.2 grade level and Martin Luther King’s “I Have a Dream” speech is at a 9.4 grade level. Most major newspapers are written at between an 11th and 14th grade level.

The degree of decline is not evenly spread between the two parties:

“Between 1996 and 2005, Republicans overall spoke at consistently 2/10ths of a grade level higher than Democrats, except for 2001, when a rare moment of national unity also seems to have extended to speaking at the same grade level. But following 2005, something happened, and Congressional speech has been on the decline since.”

Speeches

The Sunlight Foundation study goes into detail as to what might have caused the disparity and why the length of Congressional service also makes a difference.

Readability = comprehensibility?

Of course, some think the “dumbing down” of political speech is a positive development. Simplification begets greater understanding. Or does it? It may also be easier to speak about simpler “stuff” more simply. Perhaps somewhat surprisingly, another study found that President “Obama’s SOTU addresses have the lowest average Flesch-Kincaid score of any modern president”:

With three addresses under his belt, President Obama has the lowest average Flesch-Kincaid score for State of the Union addresses of any modern president. Obama’s average grade-level score of 8.4 is more than two grades lower than the 11.1 grade average for the other 67 addresses written by his 12 predecessors.

President Obama says, “My message is simple.” Our problems remain complex. Our populace’s willingness to listen to and ability to parse the message continues to decline. Hard to say what needs to change first: speakers, what’s spoken or listeners?

Shouting

Slate iphone5

In what passes as technology journalism, 3 months = 180° turn. (Why this particular author changed his heart, brain, spleen and testosterone level for this particular story is a matter for another day.) What is worrisome here is that such fickleness of opinion has become excruciatingly common in online journalism. It pays to shout, shout first, shout often, shout loud, shout different, but most familiarly, just shout.

Shouting sells. We’ve known this for a long time. If companies are daft enough to let their ad buyers talk them into spending money on those who shout the most, then publishers would be reckless to leave money on the table. Some publishers say they would like to steer their publications away from yellow journalism, but in a compensation system based solely on pageviews and clicks, they are beholden to a Romneyesque principle: “We’re not going to let our campaign be dictated by fact-checkers.”

It’s far less important how one author feels about the iPhone 5 than the alarming fact that Slate let this author publish a 1,200-word essay about a device he hadn’t used, nearly three months before it shipped. Why? Because shouting creates pageviews and clicks, and…well, there’s nothing more to say: shouting sells. If this author or another wants to be in the game, sooner than later, he or she will have to start shouting, louder and louder.

Paradoxically, some of the most thoughtful people around work in journalism. And yet all efforts of transition from print-based to online publishing without reliance on pageviews and clicks have essentially flopped. The current crop ranges from VC-supported publicity outlets masquerading as online newsdailies to those whose contribution to civilization stop at copy-and-paste aggregation in a slide show.

While what’s new may not be fully satisfying, there’s no going back to the old either. Regardless, all around the world and especially in Europe there are calls to subsidize old print by taxing new tech:

Levy

Mind you, these aren’t really calls to incentivize companies to create new models of service delivery online but to subsidize and sustain their existing operating structures during transition to an online regime that expects them to inevitably adopt, yes, pageview advertising for survival.

Democracy

Nobody likes advertising, and yet we seem to be stuck with its corrupting effects on public discourse online. It corrupts news delivery, Facebook privacy, Twitter flow, Google search, Kindle reading and so on. There doesn’t seem to be any way to make profits online, or often just survive, without pageviews and clicks, and all the shouting that entails.

Sadly, publishing is not the only industry suffering the ravages of transition to digital. We want better and cheaper telephony, faster and more ubiquitous Internet access, digitally efficient health care, on-demand online education, 21st century banking, always-available music, TV and movies…

We believe the future is fully digital, and the future is now. And yet experimenting with new digital models not based on advertising at a scale that matter have not been successful. Entrenched players spend hundreds of millions to maintain their regulatory moats and leverage their concentrated distribution power. In Canada, just three publishing groups own 54% of newspapers. If allowed to merge, Universal and EMI would control 51 of 2011’s Billboard Hot 100 songs. Six Hollywood studios account for well over 3/4 of the market. AT&T and Verizon alone have over 440,000 employees. Predictably, the FCC remains the poster child of regulatory capture.

The un-digital camp is far from relinquishing their power. Models that can replace them aren’t here. Advertising online has been corruptive of user privacy and editorial integrity. I’m afraid it’ll be a miracle if the shouting subsides anytime soon.

Yellowcake, Yellow Journalism and Android

In 2006, Vanity Fair summarized how the Bush administration orchestrated a series of thinly disguised propaganda moves to justify the invasion of Iraq:

Thewar

The War They Wanted, the Lies They Needed

The Bush administration invaded Iraq claiming Saddam Hussein had tried to buy yellowcake uranium in Niger. As much of Washington knew, and the world soon learned, the charge was false. Worse, it appears to have been the cornerstone of a highly successful “black propaganda” campaign with links to the White House.

There were innuendos, Congressional committee “testimonies,” off-the-record briefings, “experts” on TV, Italian connections and enough subterfuge to justify a decent Hollywood movie.

But most of all, there were journalists writing and talking about it everywhere. Without the benefit of fact checking. As “the world soon learned,” the U.S. went on to invade Iraq and, regardless of what you think about that decision, the cost in human lives and treasury has been devastating.

What color is Android?

What, you may ask, does this have to do with Android? Despite the smartphone wars everybody’s talking about, no lives are at stake and nobody’s going to thermonuclear war over Android.

Here’s how the Android yellowcake was sold:

Bestselling

Amazon told us the Kindle Fire was their best selling product on Black Friday and now some early Q4 numbers have arrived. A new report from iSuppli estimates that Amazon will ship 3.9 million units before the end of the year and Digitimes says that number could climb to as high as 5 million.

In a “Report” entitled, “Amazon selling 2,000 Kindle Fires every hour” the mainstream MSNBC.com syndicated a GeekWire.com piece:

Looks like Amazon.com has a hit on its hands, even before its new tablet computer is officially released to consumers.

The Seattle company is selling pre-orders for its new Kindle Fire tablets at a rate of 2,000 an hour, or more than 50,000 per day, according to website Cult of Android, which has gotten its hands on what it describes as internal Amazon inventory documents.

If the report is accurate, and the pace continues, Amazon will have sold 2.5 million Kindle Fires prior to the Nov. 15 launch — outpacing the first month of sales for either the iPad or the iPad 2, according to the site.

which referenced a website called CultofAndroid.com which referenced “leaked” documents:

Leaked

Even the generally more responsible The Verge couldn’t help itself repeat verbatim those Kindle Fire numbers. Numbers neither it nor any other publication had, because Amazon just doesn’t bother reporting them:

Kindle Fire remains Amazon’s best-selling item, million Kindle per week sales continue

Amazon just announced that it sold more than a million Kindle devices per week throughout December — that includes the Kindle, Kindle Touch, and Kindle Fire tablet. As usual for Amazon, no specific numbers were given, but the company says the Fire remains its best-selling and most-wished-for item, marking some 13 weeks that the Android-powered seven-inch tablet has held the top spot.

Are we there yet?

Indeed, if you read about the Kindle Fire around that time in any number of print or online publications, the unmistakable impression you’d get was: the Kindle Fire was selling in record numbers. So well in fact that same journalists would soon start telling Apple to come out with a 7″ tablet of its own or lose the iPad head-start, just like it “lost” its iPhone advantage.

And yet no journalist had any concrete evidence of the yellowcake: actual Android units sold. Not from Amazon, not from Samsung, not from HTC, not from Google…Nobody has actual Android unit numbers sold, quarter after quarter, in one of the biggest and most lucrative markets anywhere, which they’re supposed to be covering. No journalist has had the gumption to ask Amazon CEO Jeff Bezos or Google CEO Larry Page, “Sir, have you no sha, err, sales numbers to give us?” And keep asking until Amazon and Samsung and HTC and Google tell us just how well they are doing. Where’s the yellowcake?

I am, of course, not the only one on Twitter who objects to this charade of corporate secrecy in the name of openness:

Bevans1

Isn’t it time honorable journalists asked Google, whose mobile operating system was created primarily to harvest users’ private info, to report just how many Android units are actually sold every quarter, lest they be labeled yellow journalists carrying Google’s water in a link-baiting game?

Store Wars: Opt out, opt in, sell out, capitulate?

Apple’s new App Store rules now mandate that users themselves must decide whether they want to give their own personal info to publishers when they subscribe. What would be the reaction of the publishing industry to this? Straight from a publisher, Forbes:

Pam Horan, publisher of the Online Publishers Association, says the trade organization’s members — a group that includes Time Inc., Hearst, Conde Nast, Bloomberg, National Geographic and, yes, Forbes — are worried the new regime doesn’t give them the flexibility they need to serve their customers.

The flexibility to serve their customers

What does Apple do to deny publishers that “flexibility” then? One click to opt in to data sharing. Pam Horan, again:

Anything that requires the consumer to take yet another step is always going to reduce the number of people that participate in the process. It limits the ability to gather audience insights to build the right products. With this inability to know who your consumers are, it really affects the ultimate product for the consumer.

Put simply, publishers don’t want readers to opt in, because they know readers will prefer to opt out. Transparency is not a friend of publishers who for decades made a mint by selling out readers to advertisers and list brokers. Most readers may not be aware of this, but those who are don’t like it. Publishers know that and hate Apple for calling their bluff. If personal info harvesting isn’t essential for publishers’ business model and it is in the interest of readers, then why would they be against an instant referendum in the form of the opt in button?

Beyond the smokescreen

This, of course, isn’t about the readers. It’s not even about Apple’s App Store. It’s about the clash of two different business models. One that sells the customer to the highest bidder through a product and the other that sells a product directly to the customer. For the former, the product is a vehicle, often an excuse, since it holds no value for the publisher. For the latter, the product is the source of value, it lives and dies by the utility and delight it brings to the customer.

Transitioning from one format or platform to a new one is often a long, arduous and financially disruptive process. Lately, however, we are seeing a time compression in these transitions. For example, moving from dial-up to broadband or from landlines to wireless took quite a bit of time. Transition from analog to digital music or from featurephones to smartphones have been much shorter. Shorter the transition, bloodier the financial impact on incumbents. Print economics have been around forever, virtually unchanged for decades. All of a sudden, though, there is an incredibly convenient format (iPad) and a platform (iTunes) for what used to live in the dead-tree zone. No wonder we have publishers up in arms about the freight train suddenly in front of them.

The grand opening

We can also look at the new App Store rules as a grand negotiation being conducted in public. Apple’s iTunes and iOS ecosystem make it abundantly clear that there’s now a platform ready for transition. Table stakes: 30% cut for the platform owner. Publishers have several choices:

1. Set up their own stores — If the most business savvy of all publishers, Rupert Murdoch, who never shied away from big and expensive bets, has come to the conclusion that News Corp alone can’t set up its own independent online store, what chance do other smaller, cash-strapped, technophobic publishers have?

2. Collude to set up a BigPublishers-only store — This would be standard operating procedure…that has repeatedly failed. Disparate corporations banding together against a successful market leader nearly always fails. Witness myriad roadkill behind iTunes.

3. Negotiate with Apple directly — Murdoch did negotiate with Apple separately, but may not have received much in return other than some technical help and launch presence. Companies like Amazon and Netflix may try to negotiate with Apple directly to leverage their popularity to wring some concessions on rules or rates.

4. Wage guerilla warfare against Apple in the press — Adobe, Part Deux. This is inevitable since many of those who produce the anti-Apple hysteria write for the publications that would financially benefit from a change in App Store policies.

5. Ask for government help — Publishers will likely ask the government to intervene and conduct a threatening investigation of App Store policies to browbeat Apple into changing its policies. Also, as the last refuge of scoundrels, they will appeal to the Congress for tax payer support under the guise of saving jobs.

6. Give up subscriptions — Google would love publishers to just give up the notion of subscriptions and go ads-only, either as free apps supported by AdMob on mobiles or browser based apps supported by AdSense. Sadly, content providers aren’t immune to making monumentally stupid mistakes and…imploding.

7. Accept Apple’s terms — We heard similar, if not identical, complaints about the size of Apple’s cut or its intermediary position between content owners and customers at the onset of iTunes and later App Store. Nobody’s complaining much about those anymore, mostly because there have been no credible alternatives.

8. Create alternatives to iTunes/iOS — This is the perennial Plan B, if Apple doesn’t budge. The usual suspects are those with a store and the will to spend money liberally to undermine Apple, namely Google, Microsoft and Amazon. Google recently transferred its upcoming music store to Andy Rubin’s Android division and is now negotiating with publishers. Microsoft rolled Nokia’s Ovi into its own store and would be happy to bankroll publishers to attract Windows Phone users. Amazon has already tangled with Apple last year after the introduction of iBooks over the agency model Apple offered to publishers. These are all big competitive players with plenty of cash to render as absurd any notion that Apple somehow has a monopoly over digital stores. It is, however, a reminder that all such previous attempts to cut down Apple by direct competition has failed.

Rock and a hard place

Apple, the one company that makes bet-the-company type moves all the time, has done it again: they have decided to cull parasitic middlemen and aggregators from the ecosystem. What choice do publishers have then? They first have to ask themselves two fundamental questions:

1. What business are we in? — Are we in the business of creating scarcity in news and media to leverage it against eyeballs for advertisers? Can our current model survive the transition to digital? Are we capable of setting up our own stores? If not, do we understand we must change our revenue streams radically? What sorts of structural and financial remodeling do we have to undergo internally to adjust to giving up 30% to Apple?

2. Quo vadis? — If our current distribution has to change, on whose digital platform will we move? Is there, in other words, an alternative to Apple App Store?

Whatever conclusions the publishing industry may arrive at, there’s one undeniable fact staring them in the face:

By next year, Apple iTunes/iOS ecosystem will have over 200 million of the most lucrative online demographics ever assembled by a company.

Apple didn’t become the world’s most valued tech company by being naive. The fact that Apple’s longstanding discipline of selling products direct to customers aligns nicely with customers’ interests of accessing a well curated, efficient, price-competitive, easy-to-use store is just the icing on the cake. Nobody else comes close. You can’t do business by ignoring the App Store.

UPDATE: In case there was any doubt that Google would step in to exploit the situation, the company introduced in less than a day after Apple’s announcement its own One Pass subscription payment plan, with a 10% cut. Google CEO Eric Schmidt: “We aren’t in this to make money, Google makes money in other ways.”