Using Web Applications for Service Discovery

One of the Web's greatest strengths is its ability to support links between websites. The original and core feature is the "anchor tag" -- the blue hyperlink that every web user learns how to interpret in the first five minutes of web exposure. As HTML and web application servers have matured as an application platform, many other ways of linking data between websites have been developed: cross-site Javascript inclusion, JSON-RPC, and HTML5 cross-document messaging, to name a few, and a plethora of server-to-server data exchange protocols. Through these technologies, it has become possible to combine data and visualizations from multiple locations into a single web experience.

The first wave of applications to use this technology dealt mostly with public data -- Google Maps was the clear exemplar of how cross-site application integration could be compelling for users and developers. In the second wave, which we are now in, the focus has been on personal data.

If we look at where these technologies have had the most success, a pattern emerges: when there is a clear platform winner which allows second-party developers to add value, adoption is quick, and new entrants find it very difficult to break into the market. When there are several platform contenders, second-party developers integrate with a few of the biggest and ask the user to pick which they want, but smaller competitors are usually ignored. When no platform leader has emerged, integration does not occur. A couple examples:

• Web Single Sign-On is a competitve landscape currently being fought by Facebook, Google, Yahoo, and Twitter. Many websites choose to display one, two, or all of these service brands, and ask the user to choose one (the so-called "NASCAR screen"). Within vertically-integrated application communities (e.g. the Facebook App platform) these solutions have been quite successful, but they have not yet achieved the goal of a general-purpose web identity system that preserves user choice while simplifying the user experience.
• Link Sharing is a hugely-fragmented space, with at least a dozen major contenders, including Twitter, Facebook, Digg, Delicious, StumbleUpon, and many others.
• Profile Access continues to underwhelm; despite numerous attempts at attribute exchange technology, most websites still require users to re-enter some piece of their profile data (name, address, phone number, etc). In may cases, security and privacy considerations have rendered potential solutions unworkable.
• Contact List Management has a few large contenders, with Yahoo, GMail, and Facebook providing the most interesting offerings, but the data models and user expectations are inconsistent.
• Application Notification is a non-category - most applications fall back on email. Within the Facebook application ecosystem, the ability to post events to the news feed was a clear early win.
• Payment Processing is fragmented and inconsistent. PayPal has a strong brand but does not dominate, and no "web wallet" technology has emerged as a clear winner. Most websites accepting user payments still require users to re-enter their payment card data. A major strength of the iOS platform is that it succesfully pulled this functionality into the platform layer.
• Photo Collection Management is siloed, with Flickr and Facebook as two major platform providers. Integration between photo processing sites and applications is spotty and frequently incomplete, and often restricted to a small number of providers.
• Media Preference Management is siloed inside the major retailers. Amazon has a large database of media consumption data, but Netflix, Apple, YouTube, Pandora, and Facebook have interesting databases of their own. Facebook's attempt at conveying media preference data automatically (through their "Instant Personalization" service) has not yet been perceived to have had a strong impact.

In the web developer community, there is a persistent hope that we can do better. If something in the web ecosystem could provide a registry of the user's services, and the APIs to those services could be standardized to a reasonable degree, we could remove much of the tedious development work that site developers have to do today, and catalyze a new wave of exciting web applications.

This hope is particularly keen for the Mobile Web. Re-entering data is painful on mobile devices, and to date mobile applications have been created inside data siloes instead of being integrated by a shared application runtime. A system that smoothly and easily integrated data across all of a user's applications would enable more, and better, mobile applications.

A quick survey of existing solutions

• Webfinger proposes to address service discovery by making any email address into a identifier, which any program can then resolve to a list of service URLs. It is potentially a universal solution, but would require the deployment of new server software across all web-facing domains. It also does not have a clear mechanism to address the privacy consideration of preventing disclosure.
• WebIntents is a recent proposal by Paul Kinlan which uses browser local storage, a shared domain, and cross-document messaging to create a registry of services. It is modeled on the "Intents" service provided by the Android OS, which is a good example of the kind of system I am proposing here.
• WebSend is a detailed proposal for a browser-local registry of service providers. The proposal has deep thinking about privacy considerations, and also proposes a mechanism to allow a service to restrict what data is made available to the calling site. It would require new browser technology to be developed before all of its ideas could be put into practice (specifically, the Uniform Message Policy).

Implementation issues aside, there is a problem with all of these solutions: users do not have a way of understanding the web service discovery task. Users do not have a consistent location to think about their web services, and are not accustomed to needing to describe them to the websites they visit.

To interact with a registry of service endpoints, the user must have a way to:

1. Know that a web application can provide services
2. Indicate that a given application is one of his or her "preferred providers"
3. When another application requests access to a service, select a provider (and perhaps interact with some UI to indicate which subresource to grant access to)
4. View their list of providers and remove providers that are no longer relevant or preferred
5. Discover what applications can provide a service, if they find themselves wanting to use an application that needs a service they don't have yet

A proposal: Using Installed Applications for Service Discovery

I have written elsewhere about Open Web Applications -- web applications, built in open HTML5 technology, which describe themselves to browsers to allow more granular user interactions. I propose that the web application manifest include an optional subelement that describes the service classes, and service endpoints, supported by an application. The application runtime can then provide a JavaScript interface to discover and exchange message messages with them.

The syntax could be, roughly:

{
	name: "My Cool Photo App",
	services: ["notification", "search", "photos"]
	service_endpoint_path: "/services/svc_endpoint"
}

The interaction model for communicating with sites would then be similar to that described in WebSend - a consuming application requests an introduction by service class, and provides a callback to be invoked when the service is found. The user agent is responsible for creating an iframe and/or window to communicate with the remote service and for relaying cross-document messages. (One concrete possibility would be for the application to simply embed an array of WebSend Providers)

This runtime could be implemented entirely in HTML5, when served by a trusted domain, by embedding hidden iframes which use cross-document messaging. Alternatively, a browser-native implementation could provide the same API with whatever user experience and security enhancements it chose to add.

The benefits of this system are:

1. There is a clear user model of where services are coming from, and a strong statement of intent: the user "installs" an application to make the services available.
2. The services are strongly bound to a user-facing brand and web experience: the user does not need to understand URLs or "web services"
3. Removing a service from a user's list is accomplished by deleting the app.
4. Sharing the discovery burden with the general web application discovery problem - application directories and search engines can easily index web applications by service class, for example.
5. Support for both online and offline, and cloud-based and local-storage applications, through a single API: an Open Web Application can store all its data in local storage, and cache all of the HTML and JavaScript locally, or keep everything on the server.

The definition of "service classes" is a standardization challenge. WebSend proposes using the existing dictionary of MIME types, and supporting wildcard matches; WebFinger uses XRD tags; WebIntents probably intends to define a registry like the dictionary of built-in Android actions. I propose to make the solution of the service class problem a non-goal for this effort, and to allow some experimentation by the web developer community before we try to lock it down.