counternotions

Adobe has been known to swing to its own unique interface tune on both Mac OS and Windows. Mac users have long complained about Adobe’s considerable disregard of Mac OS native technologies, from the pointed absence of full-fledged AppleScript in Photoshop for over a decade to the absence of 64-bit version of that app on Mac OS X when it will ship for Windows likely next year.

Adobe’s UI convergence

Adobe’s goal has always been to establish its own UI as a cross-platform, third alternative to both Mac and Windows native UIs. In the process, Adobe has not been shy to invent new UI conventions that clash with native UI patterns. A case in point is the rather funky window title/menu bar design approach in the upcoming Dreamweaver, Thermo and Fireworks apps:

It can be argued that the company has been quite successful at it. By the time Creative Suite 4 ships, Adobe will have created UI patterns and a look-and-feel shared by most of its family of desktop content creation apps that work virtually identically on both platforms.

However, when it comes to stretching the boundaries of application interface nothing comes close to Flash, Adobe’s ubiquitous web media runtime. From bizarre “Skip Intros” to enigmatic UI widgets, everyone has a “Can you believe somebody actually designed that?” moment. Well aware of this, Adobe has been steadily expanding a set of (default) UI widgets and components in its Flex framework that has come to define the look and feel of many Rich Internet Applications around the web. Of course, one doesn’t have to use the default Flash/Flex UI controls as they can be varied by creating custom skins:

UI convergence on the mature desktop market may be a given for Adobe, but the next frontier is to establish Flash Lite (Flash’s mobile version) as the premier runtime and application development platform for mobile devices, now dominated by Symbian, Windows Mobile and Java.

Nokia, for one, will be rolling out Flash Lite 3.0 on its S60 series phones this year:

Why not the iPhone?

While some version of Flash Lite is available on millions of phones, it’s conspicuously absent on the iPhone.

Many reasons have been floated for why Flash isn’t a good match for the iPhone: it’s slow, it hogs CPU cycles, it drains the battery, it crashes too often, it’s not optimized for Mac OS X and so on. As obvious as these reasons may be, even if all those technical issues could be solved tomorrow, there would still remain a huge divide between Adobe and Apple on the iPhone: who controls the UI?

Adobe’s conundrum

Adobe’s interface strategy on the desktop has been to avoid using native UI controls on the deployed OS in favor of establishing and leveraging its own unique cross-platform UI paradigm that its runtime engine can deliver consistently on multiple platforms.

Unfortunately for Adobe though, Apple is in the process of establishing the first mass-market, multi-touch platform with the iPhone and has already migrated it partially to its notebooks. Clearly, Apple (which acquired FingerWorks and its patent portfolio in 2005) has greater ambitions in establishing a broader multi-touch gesture paradigm, likely to cover other devices as well:

At its introduction, what immediately distinguished the iPhone among all other mobile devices was the pervasive and consistent application of a multi-touch UI. It was as big a jump from the WIMP (”window, icon, menu, pointer”) desktop metaphor to a direct manipulation paradigm as we’ve seen in the computer and consumer electronics industries.

Not unexpectedly, every other phone manufacturer has already lined up as the next “iPhone killer” with myriad prototypes and a few shipping products. It’s easy to see in the results that it’s quite problematic to duplicate the fluidity of the iPhone’s UI without infringing Apple’s numerous patents or creating groundbreaking advances. Most pretenders aren’t up to the task: here, for example, is a Flash Lite file that “transforms” Nokia S60 phones into iPhones:

Once you get to “secondary” pages (see video), the touch paradigm gives way to the ugly WIMP interface, which has been the common theme in the vast majority of “iPhone killer” attempts. Some competitors even tried to bypass multi-touch altogether. Samsung, for example, patented a rather impractical gesture-based UI where hand/finger gestures are tracked by a phone’s camera for interaction and navigation:

Apple: Not on my device!

With the iPhone, Apple has been slowly but surely creating a third major OS platform and UI aesthetics commensurate with a multi-touch driven, small scale mobile device. This is a company so obsessed with UI and user experience that it hasn’t yet introduced something as basic as copy and paste on the iPhone 2.0 because, as is rumored, it hasn’t quite nailed a proper UI pattern.

In this highly charged and competitive marketplace to establish the next UI paradigm for mobile devices, Apple is not about to give Adobe or any other company free reins to dilute its brand proposition by introducing cross-platform, common-denominator UIs and interaction patterns to be mingled with Apple’s carefully orchestrated multi-touch approach. So what does that leave Adobe with?

Adobe’s iPhone choices

To put Flash on the iPhone Adobe may:

• strike a deal to license Apple’s entire iPhone UI controls and interaction patterns and ship them with Flash, Flex and AIR development suites as components, much like its current default set “Halo.” Apple hasn’t yet shown any inkling that it’s willing go along with this.
• decide to duplicate the iPhone UI and ignore a legal threat from seriously irked Apple IP lawyers. (For a company that once sued Macromedia for UI infringement that would be supremely ironic.)
• leave the task of creating iPhone compatible Flash components and skins to third party partners and let them deal with the legal ramifications.
• put itself in an unenviable position of having one UI paradigm for the iPhone and another for the rest of the devices it runs on, which would mean that a non-trivial application delivered for the Flash Lite runtime would end up being (at least visually) broken either on the iPhone or on other devices. After all, even the simplest of Flash UI patterns like dragging or double-clicking won’t work on the iPhone as they have entirely different consequences on this multi-touch platform.
• abandon Flash’s long standing strategy of using its own “third-platform” UI paradigm distinct from the native look and feel of the OS it’s running on by using OS-native UI controls, thereby creating a litany of OS-specific headaches for designers and developers. After all, a UI designed for the multi-touch iPhone just wouldn’t be conveniently re-morphed to run smoothly on lesser devices and vice versa.
• encourage developers to design and produce separate versions of their Flash apps for the iPhone and other platforms, but that simply defeats the whole purpose of using Flash to deliver seamlessly compliant multi-platform apps.
• develop its own non-infringing multi-touch gesture library for Flash to either compete with Apple on the iPhone, or just ignore the iPhone and try to establish an anti-iPhone multi-touch platform. (In this regard, it may approach Google to make all or parts of its technology available to Android.)

Some of these theoretical choices are of course impractical as third-party applications cannot be legally loaded on iPhones without going through “The App Store” controlled by Apple.

Apple’s Flash choices

By being so different than other platforms, Apple’s multi-touch, direct-manipulation iPhone UI clearly upsets Adobe’s strategy of a distinct, non-native UI paradigm consistent across multiple platforms and devices. With its recent purchase of chip design house PA Semi, Apple has signaled that it’s aiming to further differentiate its iPhone platform by tying native capabilities to custom-designed system-on-chips, unavailable to third parties and cloners alike. It’s not difficult to imagine that various iPhone UI capabilities will be enhanced by the power of Apple-only hardware components. So it will be increasingly difficult for Adobe or others to battle with Apple either subversively on the iPhone or competitively on other devices.

So what about Flash on the iPhone? Can the iPhone succeed without it? That’s a two-pronged challenge for Apple.

When pundits refer to the absence of “Flash” on the iPhone, they usually mean streaming Flash video, the kind that put YouTube on the map. However, that was the easier problem for Apple to (at least partially) solve. Through an agreement with Google, Apple was able to convince the biggest conduit of Flash video online to make its videos available in the widely deployed industry-standard H.264 codec, which also drives QuickTime videos. Not too long after that Adobe switched to H.264 as its primary video codec in Flash Player 9.

The other missing part of Flash is the ubiquitous .swf runtime that delivers Rich Internet Applications in the browser, and now with Adobe AIR on the desktop as well. As we have discussed previously, unlike Adobes’ Flash, Microsoft’s Silverlight and Sun’s JavaFX, Apple doesn’t have a multimedia/RIA runtime. While all three companies have publicly expressed keen interest to put their runtime engine on the iPhone, Apple has shown no interest thus far. Indeed, despite great pundit pressure, Steve Jobs went out of his way to declare Flash “too slow to be useful” on the iPhone and Flash Lite “not capable of being used with the web.”

Apple’s “RIA runtime” is turning out to be WebKit, as we detailed previously. With the upcoming MobileMe apps Apple’s arguing that an Ajax-based UI in a web browser can be as effective as a desktop app or an RIA delivered via Flash. In other words, Apple is saying: if you want to natively deliver an app for the iPhone use Cocoa Touch and if you’re reaching for cross-platform ubiquity (including on the iPhone with Safari mobile) use Ajax. Not a complicated proposition.

As for Flash, last year an Apple developer document (”Optimizing Web Applications and Content for iPhone”) listed just one item under the section “Unsupported Technologies”:

“You’ll want to avoid using Flash and Java for iPhone content. You’ll also want to avoid encouraging users to download the latest Flash on their iPhone, because neither Flash nor downloads are supported by Safari on iPhone.”

Clearly, Adobe has an uphill battle to either re-engineer the Flash runtime for the iPhone or convince Apple to specifically accommodate it by demonstrating an indisputable need for it.

In yesterday’s Runtime wars (1): Does Apple have an answer to Flash, Silverlight and JavaFX?, we surveyed the primary contenders for rich-media, cross-platform application delivery: Flash, Silverlight and JavaFX. And wondered if Apple has an ace up its sleeve.

Apple’s options

Thanks to the surging popularity of Mac OS X and the iPhone, other companies are already bringing their plugins to Apple’s platforms, so the company may feel no urgent need to reinvent the wheel on Mac OS X. With Core Image, Core Animation, Core Audio, QuickTime and Cocoa, Apple’s desktop platform is not exactly a rich-media desert. Most of those capabilities are carried over to its post-PC devices AppleTV, iPhone and iPod touch via Mac OS X, so doing nothing may be one alternative.

Another “Apple should really…” item that refuses to die is the porting of core Apple media technologies to (at least) Windows, with a runtime for Cocoa. The “Yellow Box for Windows” which would have made write-once, deploy-anywhere possible still remains a dream of Cocoa developers.

Yet another option is #1 in Apple fans’ perennial Top Ten wishlist: buying Adobe (for Flash). This one’s quite unlikely, even during a week when Adobe’s CEO just called it quits.

Finally, Apple may choose to leverage an open source project such as OpenLaszlo, capable of outputting to multiple runtimes:

Core X

Of course, Apple could also be hard at work creating a brand new platform that integrates its Core X technologies into a new cross-platform runtime.

One huge barrier to that approach is distribution, however. It’s easy for Microsoft to distribute Silverlight through its OS monopoly. Flash, which Adobe claims is on 98% of desktops, has an unrivaled uptake rate for upgrades. Sun has close relationships with mobile device manufacturers and carriers and Java already runs on hundreds of millions of phones. While Apple claims 500,000 iTunes downloads and QuickTime is also on a lot of Windows desktops, it would still be a very difficult hurdle to get a new runtime on a broad range of platforms at this juncture. Yes, if the new platform is unambiguously superior it could happen, but Apple is coming way behind in this race.

So what then is Apple’s most sensible option? When in strategic doubt, resort to the famous stratagem devised by Odysseus.

Apple’s Trojan horse

Apple has been pretty good with the Trojan horse strategy, leveraging a few key assets (even when regarded as being late to the party) into domination in digital music, for example, and it’s trying the same with the iPhone in the mobile convergence space.

Apple’s Trojan horse in multi-platform, multimedia runtime is a piece of open source technology that’s already on Windows, Mac OS X, Linux, Adobe Flex/AIR, iPhone, iPod touch, Nokia S60 smartphones and Google’s new Android/Open Handset Alliance, with 30+ partners around the globe: WebKit 3.0.

WebKit has come a long way since Apple’s adoption of the open source KHTML rendering engine and KJS libraries from KDE. A flurry of recent additions to WebKit by Apple and WHATWG now include:

• Client-side storage via SQLite (bundled by Apple in Tiger and Leopard, but also used as a persistence layer by Google Gears and Adobe Flex) lets WebKit-based browsers store structured data locally.

• Enhanced Rich Text Editing has been a problematic area for Safari for a while, especially with Google Docs, GMail and blogging apps. WebKit 3 solves that in an elegant way by providing a fully functional rich-text editing context activated by a simple click on the editable text:

• Downloadable fonts let designers specify downloadable custom fonts via CSS @font-face rules.

• Style-able form controls are now possible via CSS specification in WebKit 3:

• Advanced CSS styling ushers in multi-column text layout, text stroke and shadow, border-radius and shadow for box effects, among others.

• Faster JavaScript and DOM parsing in WebKit 3 is said to be 2X or more, thereby speeding up rich-media applications.

• New and faster XML technologies in WebKit 3 include XPath, the W3C standard XML Path Language that lets Ajax/CSS developers query document elements more efficiently, as well as XSLTProcessor, DOMParser, XMLSerializer and much improved XMLHttpRequest.

• Scalable Vector Graphics (SVG) support for graphical presentantion and interaction via XHTML in WebKit 3 is shaping up to be the fastest among browsers. Although its rich potential has not yet been fully realized, SVG gets an opportunity to ride on the coattails of WebKit to tackle a range applications where only Flash could have been considered until now. Here’s a Mac OS X Dock-like magnification animation in SVG (WebKit 3 required for interactivity):

• CSS animation is a way of specifying animation via CSS, with transition properties like duration, timing, opacity, etc. Safari and WebKit architect Dave Hyatt promises even more:

In future posts I’ll go into transitions in more detail and also talk about another feature we’re adding: explicit animations. We are also preparing a more detailed proposal (full of intimidating spec language) that covers our thoughts on transforms, animation and other advanced visual effects.

• CSS Transforms take a page out of currently available JavaScript effects libraries to enable boxes to be scaled, rotated, skewed and translated, with timings for fades, rotation, expansion, collapses, etc. Hyatt:

…transitions are for basic implicit animations. We will also be introducing explicit animations (using actual “animation” CSS properties)…We are exploring ideas for how to do transforms in ways that could affect layout.

• HTML5 <audio> and <video> allow extremely simple media embedding in WebKit, with bi-directional JavaScript control:

<video src=sample.mov autoplay></video>

There are other additions and improvements to WebKit as well. As you can see by now, WebKit is no longer your dad’s browser. Bit by bit, it has been gaining rich-media capabilities and Apple appears to be in no mood to slow down.

Renderer on steroids

WebKit architect Hyatt outlines future directions:

Like Microsoft our engine is also used inside many applications on our home operating system. We serve two masters: the Web and OS X (in all its forms). If people need these capabilities on OS X, we are going to provide them, regardless of whether or not they end up in any specification. That said, we are very interested in standardizing this stuff, and are preparing a proposal for the folks at the CSS WG.

Also, CSS has a well-documented means of providing safe non-conflicting extensions (using vendor-specific prefixes), so there is absolutely nothing wrong with CSS extensions. This isn’t like HTML, where you end up polluting the namespace. CSS can actually be extended safely in a way that doesn’t conflict with the standards.

While Apple’s ability to set a new rich-media standard may be limited, with the open source WebKit, even competitors like Nokia, Adobe and Google are helping the company achieve its primary goal of establishing a stable and ubiquitous platform from which it cannot be strategically excluded. That’s some Trojan horse!

Is WebKit as rich and powerful as Flash, Silverlight or JavaFX? No. Certainly not today. But it’s an extremely cost-effective way for Apple to have a pervasive rich-media platform whose future it can certainly affect.

A richer version of WebKit in the near future will have a commoditizing effect on Flash, Silverlight or any other proprietary platform. Remember Apple makes money mostly by selling iPhones, AppleTVs, Macs and related services. To the extent that these devices can have access to non-propriatery runtimes, not controlled by competitors like Adobe or Microsoft, Apple would be happy to sponsor open source development, as it’s doing with WebKit.

Where to next?

Perhaps next on Apple’s agenda is beefing up its development tools for WebKit. WebKit Inspector, a sophisticated development tool to navigate and debug web apps, is already out, as is Drosera, WebKit’s JavaScript debugger. Web Clip, a seemingly simple but extremely user-friendly tool, lets non-programmers create dynamic subscriptions to a selected portion of a web page as a Dashboard widget in just a couple of steps. Here, for instance, a 48-second video of YouTube “videos being watched right now” area turned into a Dashboard widget with a single click, with the widget capable of playing selected videos without going back to a browser, all thanks to WebKit:

Of course, Dashboard widgets are themselves rooted in WebKit and Dashcode greatly simplifies the creation of these small apps. It’s not out of the realm of possibility for Dashcode or a more professional tool to eventually grow into something closer in capability to Adobe Flex Builder or other IDEs based on Eclipse.

WebKit: an open-source, license-free, rich-media capable runtime…on desktops, TVs, phones and all manner of convergence devices to come. Could it be that when Steve Jobs introduced WebKit as the way to develop for the iPhone, he actually meant to signal its future direction?

UPDATE: Today, Apple updated WebKit’s Web Inspector, with a few new features including inline CSS editing and downloaded font previews elevating it closer to a web application IDE of sorts.

Runtime competitors were outlined in Part 1:
Runtime wars (1): Does Apple have an answer to Flash, Silverlight and JavaFX?

Adobe’s got Flash, Microsoft Silverlight and Sun JavaFX. What does Apple have in this multimedia runtime war of information and entertainment delivery?

On the surface, nothing. Some might argue that QuickTime is already the answer; Flash and Silverlight are finally catching up. Further, if Apple can convince Google’s YouTube to re-encode their video inventory in QuickTime’s primary codec H.264/AVC and if the new Flash player will also feature the industry standard H.264, why bother with anything else?

Because more than just video is at stake here. Surely, both Silverlight and the latest Flash offer high-resolution video, but they also deliver (rich media) applications.

Adobe Flash

Adobe, for example, can deliver the core of a reasonably non-trivial application targeting Flash runtime intact across platforms to web browsers (Flex), desktop (AIR) and even mobile devices (Flash Lite). Ditto, eventually, for Silverlight and JavaFX.

Microsoft Silverlight

Microsoft’s Silverlight can not only play HD 720p video but also includes CoreCLR, the .NET Common Language Runtime, so that applications can be written in any .NET language and run on Windows, Mac OS X and soon Linux. While Silverlight can be seen as an extension of Windows Presentation Foundation, a .NET 3.0 layer, it also supports fast compiled/just-in-time JavaScript and Ruby, Python and VBx.

Sun JavaFX

Not to be outdone by Adobe and Microsoft, Sun recently introduced JavaFX, JavaFX Script, JavaFX Mobile. Based on Java, these technologies cover nearly the same domain as Flash and Silverlight.

Powerful runtimes

This new breed of network-aware platforms are capable of interacting with remote application servers and databases, parsing and emitting XML, crunching client-side scripts, rendering complex multimedia layouts, running animations, displaying vector graphics and overlaid videos, using sophisticated interface controls and pretty much anything desktop applications are able to do.

It is said that Adobe bought Macromedia to acquire Flash, whose runtime is likely the most ubiquitous plugin on the web. Fairly effortless to upgrade, the Flash runtime is a tremendous asset for Adobe and Flash/Flex/AIR developers to distribute their rich media applications.

Indeed the absence of Flash on the iPhone has been one of its most talked about shortcomings. Everyone expects Flash to arrive soon. Walt Mossberg at All Things Digital had said in July that the iPhone Flash plugin was imminent:

Apple says it plans to add that plug-in through an early software update, which I am guessing will occur within the next couple of months.

Yet four months and a couple of updates later, there’s no Flash support. Nor any support for Silverlight or JavaFX on any of Apple’s post-PC devices: AppleTV, iPhone and iPod touch. Is there more than the current absence of an SDK involved here? Mere Apple secrecy? NIMBY? Is Apple satisfied by merely having them on Mac OS X and not worried about its post-PC devices? Or does Apple have something up its corporate sleeve to counter this tsunami of powerful cross-platform runtimes?

The answer is in “Runtime wars (2): Apple’s answer to Flash, Silverlight and JavaFX,” here tomorrow.

A longer discussion on Core Animation and its significance is in:
What’s in Leopard for Designers (1): Core Animation