i've had a number of conversations (both online and off) about URLs, URL construction rules, URI templates, and related items. i've also been doing some research into what Leonard Richardson and i call "description formats" in our new book RESTful Web APIs. these formats are designed to describe services on the Web. they are meant to make it easier for devs working on client implementations to understand how a server works; they describe the server interface. it turns out these service descriptions almost always include URLs, too.
and that's a problem.
constraining server URLs
description formats (like WSDL, WADL, RAML, and others) often include either absolute or relative URLs. these descriptions are telling clients the exact URL for various actions. it seems like a good idea, but its not. why? because these description formats are constraining the URL space for any server that wants to implement that service. IOW, servers don't control their own URLs. there are so many variables in implementing services (operating systems, programming languages, Web frameworks, etc.) that formats which tell servers how the URLs space should be managed can sometimes run afoul of implementation details for that particular server.
and that's a bummer
hold your nose
one of the key properties of the Web is that a client should be able to "follow your nose" to locate new content and actions. IOW, clients should not memorize a set of URLs (or URL construction patterns) and should, instead, be open to discovering new links (ones that weren't there 'yesterday') and following those links (and forms) and activating them when appropriate.
but almost all description formats ignore the "follow your nose" dictum and instead tell clients to "hold their nose" and pre-build all the URLs into source code ahead of time. Essentially, these clients are built to ignore new links and forms. This keeps the client from following new paths provided by the service and basically prevents small changes over time from enriching the Web.
and that's not good.
actions, not addresses
instead of hard-coding URLs, description documents should hard-code the actions that are supported by the service (add, edit, approve, share, etc.) and describe the action details (protocol methods, arguments, etc.) and allow clients to recognize these actions and arguments and use whatever URLs appear at runtime to complete the actions. clients are then able to recognize the same actions used by other servers even when the exact address of these actions are different for each server implementation.
it's the actions that count, not the addresses.
describe the service, not the server
essentially, what most existing description formats do is describe a particular server's implementation, not the service itself. this is fine if your single implementation is so large and popular that hundreds of client devs are happy to create apps just for that single server instance. Twitter, Facebook, Google+, and others are in this group.
but what about all those hundreds of other cool servers out there doing great things on a smaller scale? what about servers like RStatus, Identi.ca, Tout, and others? and what about the 500+ shopping services registered on ProgrammableWeb? is describing each of these implementation separately the only way to go?
the habit of describing each and every server instead of a general description of a service is a wasteful habit we need to break.
there is another way....
Leonard Richardson and i have been working on an alternative approach to describing services. one that does not require constraining a server's URL space. one that describes the service itself, not the implementation of a single server. an approach that focuses on describing the actions, not the addresses. it is very early in the process, but we think ALPS makes it possible to break out of the bad habit of pre-defining URLs and frees client apps to get back to "following their nose" when interacting with a service, no matter where that service lives on the Web.
over the coming weeks, i'll talk more about the emerging ALPS spec and explore the possible ways in which breaking the straight-jacket of URL-based description can increase both the stability and evolvability of services on the Web.
ALPS (or something like it) can help us create better, more powerful service descriptions - and that's a good thing, right?
there are times when, as i am talking to some people about about Hypermedia and REST, i get the feeling i am talking to aliens. this happens when it becomes apparent that what i mean when i talk about these things is not the same thing others think i mean. it used to unnerve me. now, i am mostly intrigued and usually entertained.
when it becomes clear that we're not sharing the same basic understanding of things (e.g. REST !== HTTTP, Hypermedia is not limited to HTML, etc.) i usually stop, take a moment and then start asking probing questions. things like:
- Do you think HTTP and REST are the same thing?
- Do you think REST and CRUD are the same thing?
- Do you think HTTP and CRUD are the same thing?
- Do you think REST is a standard; that there's an RFC for it?
- Do you think links and forms can only appear in HTML responses?
- Do you think APIs cannot include Hypermedia controls in responses?
there are many other Qs, but these usually have the effect of getting folks to provide responses that help each of us focus in on a shared set of terms and assumptions that make a discussion enjoyable instead of a wrestling match.
and that process came to mind as i listened to some of Bret Victor's excellent talk, The Future of Programming. about mid-way through the talk, he references J. C. R Licklider's 1963 paper "Memorandum For Members and Affiliates of the Intergalactic Computer Network" where he speculates on how a giant network of computers will need to handle the case where two machines have "just met" and need to work out how to communicate successfully with each other.
Here's an excerpt of what has become know as "The Communicating with Aliens Problem":
Consider the situation in which several different centers are netted together, each center being highly individualistic and having its own special language and its own special way of doing things. Is it not desirable, or even necessary for all the centers to agree upon some language or, at least, upon some conventions for asking such questions as "What language do you speak?" At this extreme, the problem is essentially the one discussed by science fiction writers: "how do you get communications started among totally uncorrelated 'sapient' beings?"
basically, the two machines need to "probe" each other, sharing data back and forth and working to identify "shared understanding" that can be the basis of further communication. there are many implications and complications in this simple idea but the essence is sound: if you want to make sure to communicate successfully, you need to run some tests and establish some shared understanding.
doing things the hard way
the same problem exists for any machine-to-machine communications. there needs to be some a priori understanding, some shared information, before two machines can start. this is the way things work today. machines need to share understanding ahead of time.
the most common way to share this understanding is for machines called "servers" to expose an interface (API) and then have some human write lots of documentation that other humans can read in order to program other machines called "clients" to use the interface to get something done. Bret Victor points out in his talk that this process (human codes server, writes documentation, other humans read docs and write clients) doesn't scale. and i think he's correct.
what is needed is a way to reduce the human dependence; come up w/ ways to get the same kind of results w/o the same effort from humans. and i think that is also quite possible.
we have language models
lucky for us there is at least one network protocol that supports a number of "probe" options in ways that could help solve Licklider's problem: HTTP. this protocol has the notion of content negotiation built in. unfortunately, most HTTP devs miss the point of this feature and/or ignore it completely. by establishing a series "languages" that machines can use to speak to each other, HTTP can be used to allow machines to "probe" each other and discover which "languages" they share.
again, this "language" model already exists. HTML is one of the more powerful examples. it has the ability to do more than communicate data. it can also communicate possible actions via the use of Hypermedia affordances (e.g. links and forms). and it is not the only "language" that we have today to handle this task. some of the more recently registered languages include:
and there are other, as yet unregistered, languages machines can use to communicate not just data, but also actions. not just the "nouns" but also the "verbs" with which to share information between machines.
we have vocabularies
of course, the use of a protocol that allows for negotiating a mutual language model is just part of the solution. another important feature is the ability to share a vocabulary between parties. to use humans as an example, even if we both speak French, we may still be unable to share understanding if you are talking about medical topics and i have no idea of the subject whatsoever. to solve this, we need shared vocabularies, too.
and, of course, we have them, too. here are a few of the most commonly used vocabularies:
and RDF offers a rich model for defining and expressing new vocabularies. RDF is a meta-model for vocabularies.
the Licklider Turk
so, we have a protocol that supports negotiation, we have a set of languages machines can use to communicate, and we have a set of vocabularies that machines can use to share information using those languages over a protocol. well, have we solved the problem?
in fact, we continue to assume a kind of "mechanical turk" form of the internet where there is always a human (actually many humans) hidden in the underworks making the machines appear to be communicating. it seems like an elaborate parlor trick.
as long as we continue to rely on hidden humans to make the Web work we'll be limited by the scope and power of any single human. instead, we need to be thinking beyond the human limits and start working to reduce the dependence on humans to define and animate server interfaces (APIs) and stop the practice of using humans to create one-off custom clients that only work w/ one of the server interfaces created by some other human.
ALPS - a possible solution
over the last year, i've been puzzling over this problem. why haven't we gotten further along in building systems where machines have more freedom and more ability to negotiate interfaces and select shared vocabularies? i think the reason is that the three pillars of this process (protocol, language, vocabulary) have not yet spent enough time working together to create workable solutions. i've seen great efforts within some of these camps (REST in the protocol space, various media types in the language space, RDF in the vocabulary space, etc.). but i've not seen much evidence of successful inter-disciplinary work. maybe this is going on and i'm missing it. if so, i am looking for pointers.
in the meantime, i and Leonard Richardson have been working on a model called Application-Level Profile Semantics or ALPS. this is an attempt to capture both vocabulary and action information in a protocol- and language-agnostic manner. this allows machines to express the same information "profile" over many media types (languages) and protocols (HTTP, CoAP, WS, etc.) without losing any fidelity within the vocabulary.
this is an early effort and i am happy to hear from folks who have comments and suggestions in this space. i'm especially interested in anyone who would like to join us in working on the ALPS project. we already have a few people working with it and the feedback has been very helpful.
preparing for the Intergalactic Computer Network
there you go, from "Communicating with Aliens", through the Licklider Turk, on up to the ALPS effort to address the problem posed by the existence of an Intergalactic Computer Network. there are lots of opportunities here!
it's that time of year again! RESTFest 2013 is less than two months away (Sep 19-21) and preparations and are full swing. now in its fourth year, RESTFest has become one of my favorite events on the calendar. and i am very much looking forward to this year's event.
REST is Just the Beginning
this year the keynote will be delivered by Brian Sletten. and, judging from the title (and from my knowledge of Brian's experience and knowledge), it will be a great talk. we're honored that Brian accepted our invitation and looking forward not just to his presentation but the resulting converstations and explorations that are a hallmark of RESTFest.
an important part of RESTFest is the principle that everyone who shows up must give a presentation. the talks are typically quite short: a five minute Lightning talk followed by a short Q & A session. there are a few 30 min "featured talks", too. but the basic idea is that we all get to talk about things that are interesting to us and we don't have to make a big deal about it.
every year i learn 30+ new ideas and novel approaches to problem-solving and get to talk to the people who are coming up with these great things. RESTFest is a great boost to my creative spirit.
the corollary to our key "talk" principle is that we all get to listen, too. and listening is, IMO, even more important than speaking. RESTFest attendees come from all sorts of background, experiences, and points-of-view. the chance to hear how others view the Web space, how others are tackling problems, and how others are advancing the practice of services on the Web is always an eye-opener.
less theory, more practice
and that leads to another key aspect of the weekend. the focus is on "doing", not "theorizing." we're a decidely non-pedantic bunch and are usually much more interested in cool solutions than compelling theories. while it may still be common to think of anything with the REST acronym in the name to be a meeting of pointy-headed geeks, that's not us. each event i get to see actual code solving actual problems in the real world.
we hack, too
every year we host a hack day where everyone gets together to work on cool REST-related web stuff. this year Erik Mogensen will be leading the day. from what i've seen, he's got some cool ideas in store for us, too.
it's easy to join us
just as we cut down on the "ceremony" surrounding speaking and participating in a conference, we also try to eliminate the "ceremony" around signing up and showing up for RESTFest. it's quite easy:
- join our mailing list to see what we're all about
- drop into the IRC channel to chat us up
- hope onto the github wiki and create your People Page and tell us about yourself>
- head over to the registration page and reserve your seat for the event.
there's no waiting to see if your talk was accepted. no wondering if what you're working on would be "interesting" to some review committee. just sign up, post your ideas, and head down to sunny Greenville, SC for a great weekend.
need more RESTFest NOW?
can't wait for RESTFEst 2013 to get started? check out our vimeo channel w/ all the talks from previous years. there's lots of very cool stuff there!
see you in September.
it's a small diff, really - one of definitions - but a difference i think is worth calling out. when working in large distributed systems, it's better to aim for interoperability than integration. and here's why...
integration for a single system
The Merriam-Webster Online Dictionary has a definition for integration that speaks to my point:
...the combining and coordinating of separate parts or elements into a unified whole...
a common approach to working with large distributed systems (e.g. internal networked implementations that run at various locations within a single organization OR implementations that rely on some WWW-based service[s]) is to attempt to treat the entire operation as a single unit - a "whole system."
these "whole systems" can also be given the name "closed systems." IOW, people work to create a fully-controlled single unit that, even when separated by space (location) and time ("We built that part three years ago!"), the expectation is that things will work as if they are all physically local (on a single machine) and temporally local (there is no meaningful delay in the completion of requests). as you might expect, attempting this almost always goes badly - at least at any significant scale. IOW, "toy" systems will likely not exhibit any problems.
there are several reasons for attempting the approach. the most common is that treating everything as "your system" is mentally easy. another reason this "single system" view prevails is that most tooling acts this way. the legacy of edit and build tools is that all components and data are local and easily accessible. how else would we be able to do things like 'code completion' and data model validation?
anyway, the point here is that 'integration' is an anti-pattern on 'The Web.' it's not a good idea to use this as your mental model when designing, implementing, and deploying large-scale systems.
interoperability for working with other systems
as you might guess, i find Merriam-Webster's definition for interoperability much more valuable:
ability of a system ... to work with or use the parts or equipment of another system
the interoperability mindset takes a different approach. in this view, you want to treat as many things as possible as interchangeable; as things that can be 'swapped out' or re-purposed along the way. interestingly Merriam-Webster notes the first known use of this term was in 1977. so the idea of interoperability is relatively 'new' compared to 'integration' which Merriam's claims was first used in 1620.
an interoperability approach leads to systems that do not need to 'understand' each other, just ones that use 'interchangeable parts.' especially in widely-distributed systems, this interchangeability has a very high value. it's easier to replace existing items in a system (e.g. changing data-storage vendors), re-use existing parts for other needs (e.g. apply the same editing component built for a blogging service to a new print publishing service), and even re-purpose parts when needed (use the file-based document caching system to provide caching for logged-in user sessions).
the primary challenge to thinking like an inter-operator instead of an integrator is that there are no easy 'tools' for this kind of work. most all integration work is done by creative thinkers 'in the field' ("You know, we could just use our existing storage system for that..."). you usually need a rather specific knowledge of what's available on site and what the existing parts can do in order to execute on interoperability.
despite the extra cost of interoperability, there are important benefits for distributed systems that must operate over a long period of time. that's why so much of the WWW relies on interoperability. the standards we use for DNS, HTTP, HTML, etc. all assume that varying products and services are free to decide what they do and how they do it as long as they inter-operate with other products and services on 'The Web.'
treat the network space as a bunch of 'other systems'
if you take the approach that everything in your 'network space' (e.g. your local intranet or any system that relies on at least one WWW-based service) as a bunch of 'other systems' you'll be better of in the long term. you'll stop trying to get everyone to work the same way (e.g. use the same storage model, or object model, or resource model, etc.) and will be free to start working with other teams on how you can share information successfully across systems via interoperability.
even better, large organizations can get a big value out of creating an interoperability model for their implementations. this amounts to fostering an internal ethos where it's fine to be creative and solve problems in novel ways using whatever means are at your disposal as long as you make sure that you also support interoperability to the rest of the parts of the system. IOW, you have the freedom to build whatever is most effective locally as long as it does not threaten your interoperability with the other parts.
there are lots of other benefits to adopting interoperability as the long-term implementation goal, but i'll stop here for how. however, the upshot of all this rambling can be summed up here:
As a general rule, treat your implementations as exercises in interoperability, not integration.
in a span of about two weeks i'll visit four cities, three countries, and two continents as part of Layer 7's continuing free API Workshop series. along the way, i'll be joined in each city by great folks from both @layer7 and @CAinc.
Layer 7 has already hosted lots of these events this year in both Europe and North America with content delivered by my API Academy colleagues Ronnie Mitra, Alex Gaber, Holger Reinhardt, and Matt McLarty. over the last several months, i've had the pleasure to meet dozens of attendees working on some incredibly interesting projects using APIs on the Web and on internal networks.
each 1/2 day event includes high-level summaries of the most popular topics from our Introduction to APIs Workshop and our API Design & Architecture Boot Camp and, like all our workshops, is highly interactive. whether you are just starting to consider incorporating APIs into your distribution model or are already well into a live implementation, these sessions are a great way to see and hear how others are approaching the same space and to ask questions about how you and your organization can improve the design, implementation, and life-cycle maintenance of your Web-based APIs.
across the globe in two weeks...
- Chicago - Jul-16
- if you're in the US Midwest, there are still a few open seats for our July 16th API Workshop.
- Sydney - Jul-24
- Melbourne - Jul-25
- i'll be joined at the Sydney and Melbourne events by Layer7's CTO Scott Morrison.
- Toronto - Aug-01
- the Toronto event will include a presentation by Layer 7 co-founder and SVP Dimitri Sirota.
we're getting great feedback from attendees and these events continue to be free to all who show up. if you haven't been able to attend one of our workshops yet this year, now is a great time to pick a location near you, sign up, and see what the fuss is all about.
one more thing...
BTW - if you don't see a convienent location in this list, don't worry. we're already gearing up for our fall schedule and you'll be seeing lots of new locations and new content appearing soon.