“The Creepy Line”

When asked in 2010 about the possibility of a Google “implant,” Google’s then-CEO Eric Schmidt famously said:

“Google policy is to get right up to the creepy line and not cross it.

With your permission you give us more information about you, about your friends, and we can improve the quality of our searches. We don’t need you to type at all. We know where you are. We know where you’ve been. We can more or less know what you’re thinking about.”

Since that reassuring depiction of what awaits us in the future, Google has danced energetically around “the creepy line” many times, from subverting users’ privacy preferences in Safari and paying the largest FTC fine in history to introducing the omniscient Google Glass that gets as close to human trafficking as possible without drilling into the brain.

When the internet behemoth raises the bar, others rush to conquer and some manage to surpass it. Buried in the minutiae of CES 2013, in a booth not much smaller than a 10,000-inch Samsung UHD TV, was Affectiva, showcasing its primary product Affdex:

“Affdex tracks facial and head gestures in real-time using key points on viewers’ face to recognize a rich array of emotional and cognitive states, such as enjoyment, attention and confusion.”

affdex1

Affdex2

Deciphering concealed emotions by “reading” facial microexpressions, popularized by Paul Ekman and the hit TV series Lie To Me, is nothing new, of course. What’s lifting us over the creepy line is the imminent ubiquity of this technology, all packaged into a web browser and a notebook with a webcam, no installation required.

Affdex3

Eyes Wide Shut

Today, Affectiva asks viewers’ permission to record, as they watch TV commercials. What happens tomorrow? After all, DNA evidence in courts was first used in the late 1980s and has been controversial ever since. It’s been used to exonerate incarcerated people as well as abused and misused to convict innocent ones. Like DNA analysis, facial expression reading technology will advance and may attain similar stature in law and in other fields…Some day.

Currently, however, along with its twin brother face recognition technology, microexpression reading isn’t yet firmly grounded in law. This uncertainty gives it the necessary space to evolve technologically but also also opens the door to significant privacy and security abuse.

Janus-faced

The technology, when packaged into a smartphone, for example, can be used to help some of those with Asperger’s syndrome to read facial expressions. But it can also be used in a videotelephony app as a surreptitious “lie detector.” It could be a great tool during remote diagnosis and counseling in the hands of trained professionals. But it could also be used to record, analyze and track people’s emotional state in public venues: in front of advertising panels, as well as courtrooms or even job interviews. It can help overloaded elementary school teachers better decipher the emotional state of at-risk children. But it can also lead focus-group obsessed movie studios to further mechanize character and plot development.

The GPU in our computers is the ideal matrix-vector processing tool to decode facial expressions in real-time in the very near future. It would be highly conceivable, for instance, for a presidential candidate to be peering into his teleprompter to see a rolling score of a million viewers’ reactions, passively recorded and decoded in real-time, allowing him to modulate his speech in synchronicity with that real time feedback. Would that be truly “representative” democracy or abdication of leadership?

And if these are possible or even likely scenarios, why wouldn’t we have the technology embedded in a Google Glass-like device or an iPhone 7, available all the time and everywhere. If we can use these gadgets to decode other people’s emotional state, why can’t these gadgets use the same to decode our own and display them back to us? What happens when, for the first time in homo sapiens history, we have constant (presumably unbiased) feedback on our own emotions? The distance from detecting emotional state by machines to suggesting (and even administering) emotion altering medicine can’t be that far, can it? How do we learn to live with that?

The technology is out there. From Apple’s Siri, Google already has the blueprint to advance Google Now from searching to transactions. One would think the recent hiring of Singularity promoter Ray Kurzweil as director of engineering points to much higher ambitions. Ambitions we’re not remotely prepared to parse yet. Much closer to that creepy line.

Is Siri really Apple’s future?

Siri is a promise. A promise of a new computing environment, enormously empowering to the ordinary user, a new paradigm in our evolving relationship with machines. Siri could change Apple’s fortunes like iTunes and App Store…or end up being like the useful-but-inessential FaceTime or the essential-but-difficult Maps or the desirable-but-dead Ping. After spending hundreds of millions on acquiring and improving it, what does Apple expect to gain from Siri, at once the butt of late-night TV jokes but also the wonder of teary-eyed TV commercials?

Everyone expects different things from Siri. Some think top 5 wishes for Siri should include the ability to change iPhone settings. The impatient already think Siri should have become the omniscient Knowledge Navigator by now. And of course, the favorite pastime of Siri commentators is comparing her query output to Google Search results while giggling.

Siri isn’t a sexy librarian

The Google comparison, while expected and fun, is misplaced. It’d be very hard for Siri (or Bing or Facebook, for that matter) to beat Google at conventional Command Line Interface search given its intense and admirable algorithmic tuning and enormous infrastructure buildup for a decade. Fortunately for competitors, though, Google Search has an Achilles heel: you have to tell Google your intent and essentially instruct the CLI to construct and carry out the search. If you wanted to find a vegetarian restaurant in Quincy, Massachusetts within a price range of $25-$85 and you were a Google Search ninja, you could manually enter a very specific keyword sequence: “restaurant vegetarian quincy ma $25…$85″ and still get “about 147,000 results (0.44 seconds)” to parse from. [All examples hereon are grossly simplified.]

Linear

This is a directed navigation system around The Universal Set — the entirety of the Internet. The user has to essentially tell Google his intent one. word. at. a. time and the search engine progressively filters the universal set with each keyword from billions of “pages” to a much smaller set of documents that are left for the user to select the final answer from.

Passive intelligence

Our computing devices, however, are far more “self-aware” circa 2012. A mobile device, for instance, is considerably more capable of passive intelligence thanks to its GPS, cameras, microphone, radios, gyroscope, myriad other in-device sensors, and dozens of dedicated apps, from finance to games, that know about the user enough to dramatically reduce the number of unknowns…if only all these input and sensing data could somehow be integrated.

Siri’s opportunity here to win the hearts and minds of users is to change the rules of the game from relatively rigid, linear and largely decontextualized CLI search towards a much more humane approach where the user declares his intent but doesn’t have to tell Siri how do it every step of the way. The user starts a spoken conversation with Siri, and Siri puts an impressive array of services together in the background:

  • precise location, time and task awareness derived from the (mobile) device,
  • speech-to-text, text-to-speech, text-to-intent and dialog flow processing,
  • semantic data, services APIs, task and domain models, and
  • personal and social network data integration.

Let’s look at the contrast more closely. Suppose you tell Siri:

“Remind me when I get to the office to make reservations at a restaurant for mom’s birthday and email me the best way to get to her house.”

Siri already knows enough to integrate Contacts, Calendar, GPS, geo-fencing, Maps, traffic, Mail, Yelp and Open Table apps and services to complete the overall task. A CLI search engine like Google’s could complete only some these and only with a lot of keyword and coordination help from the user. Now lets change “a restaurant” above to “a nice Asian restaurant”:

“Remind me when I get to the office to make reservations at a nice Asian restaurant for mom’s birthday and email me the best way to get to her house.”

“Asian” is easy, as any restaurant-related service would make at least a rough attempt to classify eateries by cuisine. But what about “nice”? What does “nice” mean in this context?

A conventional search engine like Google’s would execute a fairly straight forward search for the presence of “nice” in the text of restaurant reviews available to it (that’s why Google bought Zagat), and perhaps go the extra step of doing a “nice AND (romantic OR birthday OR celebration)” compound search to throw in potentially related words. Since search terms can’t be hand-tuned for an infinite number of domains, this comes into play for highly searched categories like finance, travel, electronics, automobiles, etc. In other words, if you’re searching for airline tickets or hotel rooms, the universe of relevant terms is finite, small and well understood. Goat shearing or olive-seed spitting contests, on the other hand, may not benefit as much from such careful human taxonomic curation.

Context is everything

And yet even when a conventional search engine can correlate “nice” with “romantic” or “cozy” to better filter Asian restaurants, it won’t matter to you if you cannot afford it. Google doesn’t have access to your current bank account, budget or spending habits. So for the restaurant recommendation to be truly useful, it would make sense for it to start at least in a range you could afford, say $$-$$$, but not $$$$ and up.

Therein comes the web browser vs. apps unholy war. A conventional search engine like Google has to maintain an unpalatable level of click-stream snooping to track your financial transactions to build your purchasing profile. That’s not easy (likely illegal on several continents) especially if you’re not constantly using Google Play or Google Wallet, for example. While your credit card history or your bank account is opaque to Google, your Amex or Chase app has all that info. If you allow Siri to securely link to such apps on your iPhone, because this is a highly selective request and you trust Siri/Apple, your app and/or Siri can actually interpret what “nice” is within your budget: up to $85 this month and certainly not in the $150-$250 range and not a $25 hole-in-the wall Chinese restaurant either because it’s your mother’s birthday.

Speaking of your mother, her entry in your Contacts app has a custom field next to “Birthday” called “Food” which lists: “Asian,” “Steak,” and “Rishi Organic White Tea”. On the other hand, Google has no idea, but your Yelp app has 37 restaurants bookmarked by you and every single one is vegetarian. Your mother may not care, but you need a vegetarian restaurant. Siri can do a proper mapping of the two sets of “likes” and find a mutually agreeable choice at their intersection.

So a simple search went from “a restaurant” to “a nice Asian vegetarian restaurant I can afford” because Siri already knew (as in, she can find out on demand) about your cuisine preference and your mother’s and your ability to pay:

Restaurant chain

Mind you, all these series of data lookups and rule arbitrations among multiple apps happen in milliseconds. Quite a bit of your personal info is cached at Apple servers and the vast majority of data lookups in third party apps are highly structured and available in a format Siri has learned (by commercial agreement between companies) to directly consume. Still, the degree of coordination underneath Siri’s reassuring voice is utterly nontrivial. And given the clever “personality” Siri comes with, it sounds like pure magic to ordinary users.

The transactional chain

In theory, Siri’s execution chains can be arbitrarily long. Let’s consider a generic Siri request:

Check weather at and daily traffic conditions to an event at a specific location, only if my calendar and my wife’s shared calendar are open and tickets are available for under $50 for tomorrow evening.

Siri would parse it semantically as:

Chain1

and translate into an execution chain by apps and services:

Chainarrow

Further, being an integral part of iOS and having programmatic access to third party applications on demand, Siri is fully capable of executing a fictional request like:

Transfer money to purchase two tickets, move receipt to Passbook, alert in own calendar, email wife, and update shared calendar, then text baby sitter to book her, and remind me later.

by translating it into a transactional chain, with bundled and 3rd party apps and services acting upon verbs and nouns:

Chain4

By parsing a “natural language” request lexically into structural subject-predicate-object parts semantically, Siri can not only find documents and facts (like Google) but also execute stated or implied actions with granted authority. The ability to form deep semantic lookups, integrate information from multiple sources, devices and 3rd party apps, perform rules arbitration and execute transactions on behalf of the user elevates Siri from a schoolmarmish librarian (à la Google Search) into an indispensable butler, with privileges.

The future is Siri and Google knows it

After indexing 40 billion pages and their PageRank, legacy search has largely run its course. That’s why you see Google, for example, buying the world’s largest airline search company ITA, restaurant rating service Zagat, and cloning Yelp/Foursquare with Google Places, Amazon with Google Shopping, iTunes and App Store with Google Play, Groupon with Google Offers, Hotels.com with Google Hotel Finder…and, ultimately, Siri with Google Now. Google has to accumulate domain specific data, knowledge and expertise to better disambiguate users’ intent in search. Terms, phrases, names, lemmas, derivations, synonyms, conventions, places, concepts, user reviews and comments…all within a given domain help enormously to resolve issues of context, scope and intent.

Whether surfaced in Search results or Now, Google is indeed furiously building a semantic engine underneath many of its key services. “Normal search results” at Google are now almost an afterthought once you go past the various Google and third party (overt and covert) promoted services. Google has been giving Siri-like answers directly instead of providing interminable links. If you searched for “Yankees” in the middle of the MLB playoffs, you got real-time scores by inning, first and foremost, not the history of the club, the new stadium, etc.

Siri, a high-maintenance lady?

Google has spent enormous amounts of money on an army of PhDs, algorithm design, servers, data centers and constant refinements to create a global search platform. The ROI on search in terms of advertising revenue has been unparalleled in internet history. Apple’s investment in Siri has a much shorter history and far smaller visible footprint. While it’d be suicidal for Apple to attack Google Search in the realm of finding things, can Apple sustainably grow Siri to its fruition nevertheless? Very few projects at Apple that don’t manage to at least provide for their own upkeep tend to survive. Given Apple’s tenuous relationship with direct advertising, is there another business model for Siri?

By 2014, Apple will likely have about 500 million users with access to Siri. If Apple could get half of that user base to generate just a dozen Siri-originated transactions per month (say, worth on average $1 each, with a 30% cut), that would be roughly a $1 billion business. Optimistically, the average transaction could be much more than $1 or the number of Siri transactions much higher than 12/month/user or Siri usage more than 50% of iOS users, especially if Siri were to open to 3rd party apps. While these assumptions are obviously imaginary, even under the most conservative conditions, transactional revenue could be considerable. Let’s recall that, even within its media-only coverage, iTunes has now become a $8 billion business.

As Siri moves up the value chain from its original CLI-centric simplicity prior to Apple acquisition to its current status of speech recognition-dictation-search to a more conversationalist interface focused on transactional task completion, she becomes far more interesting and accessible to hundreds of millions of non-computer savvy users.

Siri as a transaction machine

A transactional Siri has the seeds to shake up the $500 billion global advertising industry. For a consumer with intent to purchase, the ideal input comes close to “pure” information, as opposed to ephemeral ad impression or a series of search results which need to be parsed by the user. Siri, well-oiled by the very rich contextual awareness of a personal mobile device, could deliver “pure” information with unmatched relevance at the time it’s most needed. Eliminating all intermediaries, Siri could “deliver” a customer directly to a vendor, ready for a transaction Apple doesn’t have to get involved in. Siri simply matches intent and offer more accurately, voluntarily and accountably than any other method at scale that we’ve ever seen.

Another advantage of Siri transactions over display and textual advertising is the fact that what’s transacted doesn’t have to be money. It could be discounts, Passbook coupons, frequent mileage, virtual goods, leader-board rankings, check-in credits, credit card points, iTunes gifts, school course credits and so on. Further, Siri doesn’t even need an interactive screen to communicate and complete tasks. With Eyes Free, Apple’s bringing Siri to voice controlled systems, first in cars, then perhaps to other embedded environments that don’t need a visual UI. Apple having the largest and the most lucrative app and content ecosystem on the planet with half a billion users with as many credit card accounts would make the nature of Siri “transactions” an entirely different value proposition to both users and commercial entities.

Siri, too early, too late or merely in progress?

And yet with all that promise, Siri’s future is not a certain one. A few potential barriers stand out:

  • Performance — Siri works mostly in the cloud, so any latency or network disruption renders it useless. It’s hard to overcome this limitation since domain knowledge must be aggregated from millions of users and coordinated with partners’ servers in the cloud.
  • Context — Siri’s promise is not only lexical, but also contextual across countless domains. Eventually, Siri has to understand many languages in over 100 countries where Apple sells iOS devices and navigate the extremely tricky maze of cultural differences and local data/service providers.
  • Partners — Choosing data providers, especially overseas, and maintaining quality control is nontrivial. Apple should also expect bidding wars for partner data, from Google and other competitors.
  • Scope — As Siri becomes more prominent, so grow expectations over its accuracy. Apple is carefully and slowly adding popular domains to Siri coverage, but the “Why can’t Siri answer my question in my {esoteric field}?” refrain is sure to erupt.
  • Operations — As Siri operations grow, Apple will have to seriously increase its staffing levels, not only for engineers from the very small semantic search and AI worlds, but also in the data acquisition, entry and correction processes, as well as business development and sales departments.
  • Leadership — Post-acquisition, two co-founders of Siri have left Apple, although another one, Tom Gruber, remains. Apple recently hired William Stasior, CEO of Amazon A9 search engine, to lead Siri. However, Siri needs as much engineering attention as data partnership building, but Stasior’s A9 is an older search engine different from Siri’s semantic platform.
  • API — Clearly, third party developers want and expect Apple someday to provide an API to Siri. Third party access to Siri is both a gold mine and a minefield, for Apple. Since same/similar data can be supplied via many third parties, access arbitrage could easily become an operational, technical and even legal quagmire.
  • Regulation — A notably successful Siri would mean a bevy of competitors likely to petition DoJ, FTC, FCC here and counterparts in Europe to intervene and slow down Apple with bundling/access accusations until they can catch up.

Obviously, no new platform as far-reaching as Siri comes without issues and risks. It also doesn’t help that the two commercial online successes Apple has had, iTunes and App Store, were done in another era of technology and still contain vestiges of many operational shortcomings. More recent efforts such as MobileMe, Ping, Game Center, iCloud, iTunes Match, Passbook, etc., have been less than stellar. Regardless, Siri stands as a monumental opportunity both for Apple as a transactional money machine and for its users as a new paradigm of discovery and task completion more approachable than any we’ve seen to date. In the end, Siri is Apple’s game to lose.

Apple’s design problems aren’t skeuomorphic

From Tim Cook’s letter announcing the latest reorganization at Apple last week:

Jony Ive will provide leadership and direction for Human Interface (HI) across the company in addition to his longtime role as the leader of Industrial Design. Jony has an incredible design aesthetic and has been the driving force behind the look and feel of our products for more than a decade. The face of many of our products is our software and the extension of Jony’s skills into this area will widen the gap between Apple and our competition.

Sir Jony

Sir Jony Ive needs no introduction

Ive’s industrial design work has been one of the key drivers of Apple’s rebirth. His relentless, iterative focus on simplification of form and function under an aesthetic budget is legend. From Bondi blue iMac to iconic iPods to flatscreen iMacs to iPhones to iPads, his signature is unmistakable.

What’s not publicly known is Ive’s role, if any, on Apple software. The current meme of Ive coming on a white horse to rescue geeks in distress from Scott Forstallian skeuomorphism is wishfully hilarious. Like industrial design of physical devices, software is part form and part function: aesthetics and experience. Apple’s software problems aren’t dark linen, Corinthian leather or torn paper. In fact, Apple’s software problems aren’t much about aesthetics at all…they are mostly about experience. To paraphrase Ive’s former boss, Apple’s software problems aren’t about how they look, but how they work. Sometimes — sadly more often than we expect — they don’t:

  • Notifications, dark linen background or not, is woefully under-designed.
  • Six items that drain mobile device batteries (GPS, WiFi, cellular radio, Bluetooth, notifications and screen brightness) still require laborious, multiple clicks in multiple places, not immediately obvious to non-savvy users to turn on and off, without any simple, thematic or geo-fenced grouping.
  • iCloud-desktop integration and direct file sharing among Apple devices are circuitous and short of “It Just Works.”
  • Many Apple apps, like the iWork suite, are begging to be updated. Others, like Preview, TextEdit, Contacts, desperately need UI and UX overhauls.
  • Core functionalities like the Dictionary or the iOS keyboard layout and auto-correction are not the best of breed.
  • iOS app organization in tiny “folders” containing microscopic icons on pages without names borders on illegible and unscalable.
  • Navigating among running iOS apps (inelegant and opaque for users) and data interchange among apps in general (vastly underpowered for developers) remain a serious problem.

Obviously, it’s not much use piling up on this list, as everyone else’s list of “things to be improved” is likely ten times longer. Neither is it really useful arguing at this point whose fault it is. Apple software — especially its self-declared future, iOS — needs some serious overhaul both in aesthetics and experience, and far more in the latter department.

One Man. One Company. One Aesthetics?

The question is, can one person, even the world’s most eminent industrial designer, pull it off? Is it possible for one person to have enough time in a day to pay sufficient attention to hardware and software — aesthetics and experience — at a level of detail that has become necessary?

After all, Apple’s Human Interaction Guidelines (HIG) has never been just about the aesthetics of icon shadows or button alignment, but also about the behavioral aspects of application design in general. A generation ago, especially prior to the ascendency of web design, HIG was far more respected and adhered to both by Apple itself and its developers. Loyal users also noticed deviations and complained. HIG debates on public forums were not uncommon.

Today, not so much. A radio button or a checkbox can initiate a webpage navigation. Navigational menus now come in circular and triangular popups. There are now purely gestural UIs otherwise betraying none of their functionality to the user. Layers of sliding panels cascade on each other. List items slide left and right to initiate drill-down actions, up and down to reveal interactive media. Some UIs are beveled in 3D, some flat with no shadows, most a loose melange of many styles. And with each such “innovation” they bury the notion of a once powerful HIG a foot deeper.

Is it possible then to have a Human Interface czar for a 500-million user ecosystem today at all? If it were possible, would it be desirable? And if it were possible and desirable, can one person be in charge of both the visual aesthetics and the functional experience of such a huge ecosystem?

  • Can one person truly understand how Siri’s problems surfaced at the display layer actually go deeper into its semantic underpinnings, phoneme parsing, lexical contextuality, data-provider contracts, network latencies, etc., and therefore how the overall solution depends on this interplay of form and function?
  • Is it fair and reasonable to expect one person to understand how user-facing issues that surface within Maps or Passbook apps also suffer from similar technical and operational constraints?
  • Or how a lack of a social layer in Game Center or a content discovery layer in iTunes or App Store impede their functions in so many ways?
  • How about the cognitive mess that is document management and sharing in iClouds, as Apple moves away from user-level file management?
  • Or that monumental experiment also known as the Grand iTunes Redesign that’s been threatening to arrive any year now?
  • Or the Apple TV that needs an injection of a barrel of aesthetics and experience redesign?

In just how many UI corners and UX paths can a HI czar discover the depth of lingering problems and craft solutions as Apple designers play chicken with OS X menubar and iOS status bar translucency and color with every update? These are not just, or even mostly, aesthetic problems.

Apple, quo vadis?

It’s not known if Ive’s is a transitionary appointment necessitated by Scott Forstall’s departure or a harbinger of a longer term realignment of Apple design under a single umbrella. Unification of hardware and software design under a czar may certainly bring aesthetic efficiencies but it can also be pregnant with dangers. Much as the “lickable” Aqua UI ended up doing a decade ago, a serious mistake would be to hide many of these behavioral, functional and experiential software problems under a more attractive, aesthetically unifying display layer, such as:

  • A more modern, less cheesy Game Center redesign that still doesn’t have a social layer.
  • An aesthetically unified iTunes without appreciably better content discoverability.
  • A Siri app without the background linen but still lacking much deeper semantic integration with the rest of the iOS.
  • A Maps app without the ungainly surreal visual artifacts but still missing a robust search layer underneath.
  • An iBooks app without the wooden shelves or inner spine shadow, but still with subpar typography and anemic hyphenation and justification.
  • A Podcast app without the tape deck skeuomorphism, but with all the same navigational opaqueness.

In the end, what’s wrong with iOS isn’t the dark linen behind the app icons at the bottom of the screen, but the fact that iOS ought to have much better inter-application management and navigation than users fiddling with tiny icons. I’m fairly sure most Apple users would gladly continue to use what are supposed to be skeuomorphically challenged Calendar or Notebook apps for another thousand years if Apple could only solve the far more vexing software problems of AppleID unification when using iTunes and App Store, or the performance and reliability of the same. And yet these are the twin sides of the same systems design problem: the display layer surfacing or hiding the power within or, increasingly, lack thereof.

Yes, unlike any other company, we hold Apple to a different standard. We have for three decades. And we have been amply rewarded. If Apple’s winning streak is to continue, I hope Jony Ive never misplaces his Superman cape behind his Corinthian leather sofa…for he will need it.

Apple Maps: The Next Turn

Maps have changed.

Cartography used to be fairly simple and largely a novelty:

Portolano

Unimaginable to the users of that Genoese world map from 1457, today’s maps are used daily by hundreds of millions of ordinary people around the globe to accomplish what’s now regarded as pedestrian tasks, like 3D flyovers:

Flyover nyc

Indeed, in the post-PC era maps have ceased to be cartographic snapshots of the Earth’s terrain and become spatial portals to a vast array of virtual services:

  • Wayfinding — In the not-too-distant future, the principal feature of maps may no longer be wayfinding. Yes, we still want to go from A to B and know where B is and what’s around it. Today, however, we also want to know not just what, but who among our social network is around B, even before we get there. And we want to know not just how to get to B, but by what modalities of transportation, when, how fast, etc.
  • Discovery — Knowing “what’s where” is clearly no longer enough. We want to see 2D, 3D, panoramic and synthetically composited photographs taken yesterday (and 50 years ago to compare) sprinkled around the map. We want to see strangers’ virtual scribblings and audio recordings left at particular coordinates. We want to know where our favorite brand of coffee or most scenic bike path may be located. We want to read all the news and tweets around a dropped pin. We want locals facts accessed on a map even before we asked for them.
  • Commerce — Today we want our maps to not only show us places, but also let us shop directly. We want to tap a restaurant on a map, get its ratings and book a reservation right then and there. We want geo-fencing to alert us automatically as the GPS tracker on our map gets near a commercial point of interest, show us a discount coupon even before we walk in and get ready for a quick transaction. We want a virtual check-in on a map converted into real-life currency discount at a retailer.
  • Integration — We’re no longer satisfied with “cartography as content”. Our maps are now intention harvesting funnels. When we ask Maps via Siri a very simple question like “How is the traffic?” we’d love for her to know 1) we have a trip planned to see 2) our mother in 3) two hours, and we usually take 4) a particular route and we’re really asking about the traffic conditions 5) at our mother’s, so that we can get 6) alternate routes if it’s unusually busy. Maps without deep semantic correlation (public: directions, routing, traffic, and private: calendar, contacts, navigation history, etc.) are not part of the future we want.
  • Entertainment — This is no longer the 14th century or the 20th, so we want to experience places without being there. We want our maps to talk to us. We want to virtually fly over not just our neighborhood, but places we may never visit. We want to tour inside malls, stores and offices without moving off our couch. We want to submerge and commune with sea turtles — all within a “map” on a tiny computer we hold in our hand.

Tinker, Tailor, Soldier, Not A Mapmaker

We could go on listing just how profoundly our expectations of what a map is have changed and will continue to expand. Apple understands this more than most companies, but Apple hasn’t been a mapmaker and knows it. Five years ago, Steve Jobs himself favored partnering with companies like Google, for search and mapping backend services:

Jobspartnering

Jobs wasn’t likely thrilled to have to rely on Google or any other company for these services, but others had a significant head-start in digital maps, and Apple had its hands full reinventing the phone at the time. The intervening five years brought Apple unprecedented success with the iPhone but also a well known systems design problem: it’s very hard to change user habits and expectations once set in. Due to contractual circumstances now being breathlessly analyzed in the media, Apple finds itself having to part with an app powered by a one-time partner, now its chief rival. Regardless, users are rarely comfortable with the loss of what’s familiar, no matter how rationally justifiable the reasons might be. Enter, Mapgate.

Is old new again?

In his open letter on Maps, Tim Cook positioned Apple’s new mapping effort as “Maps 2.0″:

We launched Maps initially with the first version of iOS. As time progressed, we wanted to provide our customers with even better Maps including features such as turn-by-turn directions, voice integration, Flyover and vector-based maps. In order to do this, we had to create a new version of Maps from the ground up.

In other words, Apple seems to be not so much reinventing Maps, as evolving it into parity with its now more feature-rich cousin on Android, and, this time, without Google’s help — a tall task, given Google’s eight-year head start. In this seemingly catch-up game, rapidly increasing accuracy and coverage appear to be Apple’s first order of business.

Mapbusters, who you gonna call?

A company in Apple’s current predicament could have followed a number of curative paths. It could have hired the equivalent of more than 7,000 mapping related personnel Google is said to employ to gather, analyze and correct data. However, for other than its retail stores, Apple has no history of hiring so many personnel (8% of its entire head count) for such a narrow operation.

Unlike Google (Big Table), Facebook (Cassandra), Yahoo (Hadoop), Amazon (Dynamo) and others that have accumulated big data storage, processing and operational expertise, Apple’s not been a magnet for data scientists or web-scale infrastructure, automation, real-time analytics and algorithm design professionals. Facebook, for example, can bring online tens of thousands of servers in less than a month in an automated fashion, Apple continues to lag, underperform and underwhelm in this area.

Instead, Apple could acquire a mapping company. Unfortunately, there aren’t a lot of those around. Neither does Apple have a history of buying companies just to get process oriented employees. It’s telling that Apple hasn’t bought any of the companies it currently gets mapping data from, like Tom Tom or Waze. Further, Apple uses multiple map data sources abroad such as AutoNavi in China.

Apple could augment its accuracy efforts by crowdsourcing corrections through OpenStreetMaps, which it’s already been using elsewhere. But OSM may not scale as fast as Apple would like and, more importantly, may pose licensing issues in the future. Another avenue for Apple is to get much more aggressive and encourage a hundred million iOS 6 Maps users to actively send map corrections and suggestions to earn accumulating incentives such as App Store or iTunes credits, iOS device prizes, free trips and so on.

But these acquisition or incentive based approaches are ultimately process oriented remedies not in Apple’s DNA. You can expect, say, Microsoft wanting to code for, test and manage thousands of models, peripherals, drivers and legacy bug exceptions for Windows as they have done for a couple of decades…Apple not so much.

Of course having good map data by itself is not good enough. Apple has to decide if it really wants to clone feature by feature what has become very familiar to its own Maps 1.0 users. That is, would Apple really want to spend all its time, resources and focus to clone Google Maps (on Android) because some of its most vocal users are asking back what was degraded in iOS 6 Maps?

Playing by its rivals’ rules hasn’t been Apple’s modus operandi. Apple often enters a semi-mature industry underserved by system design and rearranges constraints and possibilities to create unique products. More recently, for example, everyone has been busy adding NFC chips to smartphones to rejigger the entire payment industry, full of entrenched players. Apple remained agnostic on payment modalities and ignored NFC, but reimagined everything else around payments: rewards, promotions, cards, tickets, coupons, notifications…all wrapped in a time-and-location based framework, thereby opening up new avenues of growth, integration and deeper ties to users in the form of its new app Passbook. In the same vein, can Apple reimagine and redefine what mobile “mapping” ought to be?

Horizontal

Fortuitously, Apple has another systems design problem in its hands, not unlike Maps. If Maps has become the gateway to mobility, iTunes has been Apple’s portal to content. iTunes started as a single-screen song organizer. On the desktop, it’s still a single-screen app, but has grown to cover the storage, backup, authentication, transaction, discovery, promotion, browsing, preview, playback, streaming, conversion and sharing of every imaginable modern media format. It’s the focal point of a quarter trillion dollar media empire. In the process, the cognitive load of using iTunes has become considerable, not to mention the app’s performance, reliability and myriad other problems. Many users complain about it. Apple’s response has been to separate various functions into individual apps: Podcasts, iTunes U, Music, Videos, Trailers, iBooks, App Store, etc.

Icons6

Developing and delivering map services as separate apps would prevent the immaturity of one or more components from bringing down the overall perception of the entire Maps platform. Can the Maps app be sliced into 8-10 separate apps: satellite, roads/traffic, mass transit, turn-by-turn direction, 3D/flyover, search, discovery, points of interest and so on? While this may make logical sense, not all users will be happy exchanging an integrated app where especially the novice user can find everything in one place for several single-purpose apps. It can get complicated. For example, millions of people commute daily to New York City from many smaller cities and at least four states, some driving through three states twice a day. Would they want to manage various aspects of that process in multiple apps?

Vertical

Clearly, Apple has already started to think about and experiment with unorthodox displays of map data, as exemplified by its “Schematic Maps” patent application. So, for instance, instead of slicing Maps into separate apps horizontally, could it be another option to display metadata vertically as layers, like the tissue-paper overlays of yesteryear?

Overlay

Conceptually, this can be an effective way to deal with complexity, data density and integration issues within one app. PlaceBase, one of the mapping companies Apple has acquired in the last couple of years, was known exactly for such layering of data through its PushPin API.

Apple could even parallelize and greatly accelerate its Maps development by starting a mini “Map Store” and actively enlisting third parties to develop layers on top of Apple’s “base-map”. Users could make nominally priced in-app purchases (think $0.99) to add custom layers to replace and/or augment Apple’s own layers. It would be very attractive for third parties, as their layers, floating on Apple default base-map, could quickly capture a massive user base in the millions (think getting 70% of $0.99 from 10-20 million users). Wouldn’t you pay $0.99 for a Google Search layer that pre-processes a local point of interest overlay?

Layered maps don’t, of course, come without issues. It would be sensible to make layers adjustably translucent so multiple layers can be co-visible and interact with each other, such as traffic and mass transit. However, too many layers could become hard to manage for novice users, simple show/hide checkboxes may not suffice. Memory management of layers in terms of pre-caching, loading and state-saving, as well as intra-layer version compatibility and licensing could be problematic. Apple would have to carefully select and test a rather limited number of vitally important layers to go on top of its base-map.

And yet a mini Map Store could help Apple catch up and pass Google Maps or other players in the field much faster than Apple alone could, as well as open up significant opportunities for Apple’s developer ecosystem.

Does Apple have to lose for Google to win?

Not if it doesn’t play by the same rules. After all, it’s Google’s game to lose. Tim Cook telling customers to use other companies’ mapping products must have taken some guts at Cupertino. It’s perhaps a conviction on his part (with his inside knowledge) that Apple can and over time will do better than the competing apps he so forthrightly listed. That’s the confident view Google would likely prefer to fly over.

To Map Or Not To Map

For over a decade, Microsoft — the monopolist of its era — treated its customers on Macs as second-class. Its Office suite never achieved parity with its Windows sibling, even when the differences were not dictated by platform architectures. Whether it was document compatibility, font-metrics, macros, integration with other Microsoft software or myriad other gotchas, Mac versions were always lacking. Every new version of Office promised better compatibility but never really delivered it. Worse, Microsoft never quite integrated Apple-grown technologies into Office to better blend it into the Mac ecosystem, claiming it would break cross-platform compatibility with the Windows versions.

Wpxo

Sadly, this wasn’t an occasional inconvenience but a source of daily frustration for millions of paying customers, corporations and individuals alike. With business so dependent on Office, Microsoft’s message was loud and clear: if you want the real thing switch to Windows.

Sufficiently annoyed by all the trouble, some users did.

Most didn’t, and haven’t forgiven Microsoft ever since.

Therein lies a lesson for Google

Today, we know that the iOS version of Google Maps has been inferior to its Android sibling for sometime, turn-by-turn directions being the most obvious example. Clearly, Google and Apple have far better ability to integrate apps into their own respective mobile ecosystems, but we don’t really know the contractual, commercial and even technical considerations for the disparity in this particular case.

So far this has not been a major issue. In the future, it may be a different matter. In Apple’s Feud With Google Is Now Felt on iPhone, New York Times says:

But [Google] would not say whether it was building an iPhone app for users to download. Its only public statement on the matter has been vague: “Our goal is to make Google Maps available to everyone who wants to use it, regardless of device, browser, or operating system.”

Google could decide not to build an app, as a gamble that iPhone users depend on its maps so much that they might switch to Android. [Emp. added]

After outsmarting and outspending its then-chief rival MapQuest, Google has been dominating mobile maps on phones for half a decade. From a corporate rivalry standpoint, Google is in an enviable position at the moment, certainly amused by the current kerfuffle around the iOS 6 Maps app.

Museo

However, it would be monumentally myopic of Google management to “decide not to build an [iOS native map] app” or to think Apple Maps’s lack of polish will cause any meaningful migration of iOS users to Android or that Apple management would let that happen. In the end, Google management may be hypocritical but, unlike their loud amen corner in the Comments section, they’re likely not blind to obvious realities.

Samsung may have become the biggest smartphone seller by volume, but I’m sure even Andy Rubin realizes Samsung doesn’t have 400+ million users (each with a credit card account) who have proven themselves to be the world’s most lucrative online demographics. These users have invested billions in Apple’s media and app ecosystem. They are the happiest bunch whenever product satisfaction surveys are taken. They upgrade regularly. They are loyal. They frequent Apple Stores with alarming regularity. They wait in line, rain or shine. They are not going anywhere.

There’s, of course, the other side of the coin. Reports argue that Google makes much more from iOS than its own Android. Since I don’t believe any numbers coming out of Mountain View (not that Google ever officially discloses meaningful Android numbers anyway) no one really knows the footprint of Google Maps on Google’s balance sheet. I suspect not much. However, for Google that makes all its money from advertising, being able to harvest spatiotemporal user data to triangulate purchasing intent must be priceless.

Every time an iOS user interacts with Google Maps, directly or through other apps that use its API, Google gets extremely useful data that soothe its search and advertising pangs, tens of millions of times a day around the globe. For Google (and now Apple) maps are an input modality to discover user intent, perhaps only rivaled by command line search and social network affinity graphs.

But direct financial contribution is not the most important rationale for Google Maps on iOS. One of the key reasons why Google has better data than Apple is the fact that for many years users of Google Maps have been sending corrections to Google, which has improved its accuracy significantly. So by not submitting Google Maps to the App Store, Google would not only give up a very significant portion of its mobile revenue, but more importantly, it would self-induce a debilitating data-blindness on the world’s most lucrative mobile ecosystem.

If Apple does admit Google Maps into the App Store, iOS devices (where Apple makes its money) would be the only mobile platform that offers both Google and Apple map apps. Of course, Google could submit such an app and Apple could reject it. Then, as Jean-Louis Gassée would say, Damned If You Google, FTC‘ed If You Don’t.

Some dilemma.