mca blog

Programming the Web with Cloud9

Tue, 17 Jan 2012 11:49:00 GMT

it's official! i've signed to do a new book for O'Reilly. the working title is "Programming the Web with Cloud9" and it covers the Cloud9 browser-based IDE for building Web apps.

i've been using the IDE for several months (actually put myself on an "all c9, all the time" regime for a while) and have really been impressed. in fact, over the last sveeral weeks i've actually been bummed out when i ran into a few cases where i actually needed to use my local workstation to handle a task instead of relying on "C9 and the Cloud" to do it all. hey, it happens[g].

hardware, software, wetware

while the main focus of the book is the Cloud9 editor, it will also cover other 'cloud-based' developer tools including CouchDB, github, and Heroku and more. in addition to the actual software and hardware, i am also digging into the 'wet-ware' aspects of cloud-based programming. this includes how cloud-based dev tools affect teams, collaboration, project schedules, code quality, etc. along the way, i will have the chance to meet and interview some very smart people who are active in shaping the future of this kind of Web developer's experience.

the journey begins

i'll officially kick off the start of the new project w/ a trip to Node Summit in San Francisco next week. if you're in the area, i'd love to meet you and get your opinions/predictions regarding these (and other) 'cloud-based' development tools. who knows, maybe some of your comments will end up in the book!

'cloud-stack' programming

Wed, 04 Jan 2012 09:56:00 GMT

over the last couple months, i've been experimenting with a "new" development stack; what i call a "cloud stack." this development environment consists of a set of independent, but complimentary tools and services - all available via the Web. they make up a "full stack" for web development that is, IMO, quite enjoyable to work with. even better, i find it a very productive environment, too.

hello, cloud

i've noticed that, along with the rise of "the cloud" as an infrastructure model, there is also a corresponding development model. this model provides a cloud-based environment that includes editing, debugging, testing, source control, project mgmt, and deployment; all accessible from nothing more than a modern (i.e. HTML5-compliant) browser.

and, just as the process of virtualizing the hardware stack has acted as a distruptive technology for the back office, this emerging virtual programming environment will, i suspect, prove to be just as disruptive, too.

my current cloud stack of choice

here is my current toolkit for cloud-stack development:

Cloud9: i've really enjoyed using the Cloud9 browser-hosted code editor over the last few months. very similar (in some ways, related) to Mozilla's Bespin/Skywriter project, C9 is (for me at least) living up to the copy on it's home page: "Cloud9 is a state-of-the-art IDE that runs in your browser and lives in the cloud, allowing you to run, test, debug, and deploy applications from anywhere, anytime."
Node.js: i started using Node.js just about a year ago for a couple experiments; i've continued to enjoy working with it ever since. it's nice that Cloud9 supports runtime-debugging of Node.js, too. like the web site sez: "Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, perfect for data-intensive real-time applications that run across distributed devices."
CouchDB: along w/ using Node.js, i've started using Apache CouchDB as a data store. from the site: "Apache CouchDB is a document-oriented database that can be queried and indexed using JavaScript in a MapReduce fashion." right now i am testing two CouchDB providers: Cloudant and IrisCouch; both seem very solid.
Github: i've used a number of git-based repos lately, but continue to rely on github as my "go-to" source control tool. it helps that the Cloud9 editor also uses git locally and will let you clone any project from github easily. forking and code review are another thing that make github very enjoyable to use.
Heroku: i've recently started using Heroku as a deployment target for my completed Node.js projects ("Agile deployment for Ruby, Node.js, Clojure, Java, Python, and Scala."). again, it's easy since Cloud9 supports git-based deployment to Herkou directly from the editor interface. it took a bit of "tweaking" to get my git master properly 'prepped' for Heroku, but once that's done, incremental updates are a snap.

one of the nicer aspects of the above stack is that you can get started with each of them at no charge. IOW, it's essentially free to experiment. not bad at all.

the bigger picture

so far, this is the stack that works for me. there are a number of other products/services out there that i haven't yet tried, but basically it boils down to the same general programming set: editor/debugger, storage, source control, and deployment. i've left out things like testing and other helpers as i am just getting around to exploring those tools, too.

but, regardless of the details, i think the pattern is clear; there are quite a number of options for putting together your own 'cloud-stack' development environment. and i suspect this kind of environment will grow popular in time. along the way, i bet we'll see other (major) vendors make offerings in this space, too. when that happens i suspect the market for this kind of work will continue to grow and get even better.

have you assembled a 'cloud-stack' programming environment? if yes, let me know. i'd love to hear about other experiences.

REST : 'inverted' architecture

Wed, 28 Dec 2011 20:56:00 GMT

Here is one way I talk about HTTP and Fielding's REST model:

the way of HTTP

HTTP was designed to favor long term (measured in years/decades) run-time stability over ease of implementation. to do this, it relies on some very simple (but powerful) constraints:

every machine that operates over the WWW is constrained to a very tiny executable interface (what we know as the HTTP verbs). The details of this restricted interface are documented in a protocol that rarely changes over time and even when it does change, great pains are made to ensure these changes do not invalidate past implementations.
every message sent over that interface consists to two basic parts: control data as a set of name-value pairs (HTTP Headers) and a single self-describing message body. This body can be "described" as 1) a structured document (text/html), 2) a binary image (image/png), 3) a list of arguments (application/x-www-form-urlencoded), etc. in HTTP the message contains not just data, but also instructions on how to operate on the data and what program-flow options are valid at that particular point in time. The list of "understood" message formats are publicly registered w/ the IANA today and is meant to evolve relatively easily over time in ways that do not "break" existing implementations.
everything of importance is reachable using a universally understood address (URI). the list of addresses, while based on standards, is essentially unlimited and implementors are free to mint new ones as often as they wish and/or abandon existing addresses at any time.

the other way

implementing solutions under these constraints is usually quite different than what most programmers/architects learn in school. most training is "the reverse" of the above rules. Such that:

the list of verbs in the interface is unlimited and their details (arguments, etc.) are subject to change. most implementation effort is focused on defining , documenting, and maintaining this variable interface.
the program flow is found in code rather than in messages. usually in multiple place (clients and servers).
the list of 'addresses' is constrained to the name of the executable programs themselves which do not change as often as the interface

why the differences matter

Fielding's dissertation, while focused on the details of identifying and documenting arbitrary styles of software architecture for distributed systems, still does a very good job of documenting a single example (or 'style') of software architecture that is closely aligned with the HTTP constraints listed at the top of this post. Fielding named his example "REST".

history has shown that the HTTP protocol is a very flexible protocol and that not all implementations need to follow the example provided by Fielding in order to meet the needs of users. for example, RPC over HTTP works just fine for many cases; esp. those that do not require system stability on the scale of years/decades.

however, the more important it is for the solution to continue to operate (and evolve) over an extended period of time, the more useful are the additional constraints Fielding identified in his example and the more important it is to optimize for run-time stability over ease/speed of initial implementation.

REST: menus and styles

Wed, 21 Dec 2011 10:06:00 GMT

"What is surprising about the Shaw et al. model is that, rather than defining the software's architecture as existing within the software, it is defining a description of the software's architecture as if that were the architecture."

Roy T. Fielding

"If you think the menu 'is' the meal, you'll have to eat your words."

Alfred Korzybski

design [for error | explorable systems]

Sat, 26 Nov 2011 14:40:00 GMT

valuable guidance when designing a Hypermedia API (emphasis and brackets "[]" mine):

Think of each action by the user [of your API] as an attempt to step in the right direction; an error is simply an action that is incompletely or improperly specified. Think of the action as part of a natural, constructive dialog between the user [and/or user-agent] and the system [your API]. Try to support, not fight, the user's responses. Allow the user [and/or user-agent] to recover from errors, to know what was done and what happened, and to reverse any unwanted outcome. make it easy to reverse operations; make it hard to do irreversible actions. Design explorable systems [APIs].

Donald A. Norman in The Design of Everyday Things, 1990.

designing (and implementing) Hypermedia APIs is more than simply exposing server-side operations to clients (users, user-agents, etc.). instead, good Hypermedia APIs include the proper amount of affordances and constraints which allow users|user-agents to successfully interact with servers in order to complete one or more tasks in an effort to accomplish a pre-determined goal.

usability (in all its commonly-known forms and interpretations) is an essential aspect of Hypermedia. usability is, in the end, an essential aspect of all systems; esp. 'complex' ones.

how usable is your API?

affordances and constraints

Fri, 18 Nov 2011 00:55:00 GMT

How can design signal the appropriate actions? ... One important set of signals comes through the natural constraints of objects, physical constraints that limit what can be done. Another set of signals comes from the affordances of objects, which convey messages about their possible uses, actions, and functions. ... Affordances suggest the range of possibilities, constraints limit the number of alternatives. The thoughtful use of affordances and constraints together in design lets a user determine readily the proper course of action, even in a novel situation.

The Design of Everyday Things by Donald Norman, pg 82

"The value of a well-designed object..."

Tue, 01 Nov 2011 10:07:00 GMT

"The value of a well-designed object is when it has such a rich set of affordances that the people who use it can do things with it that the designer never imagined."

Donald Norman, 1994

hypermedia binding

Tue, 04 Oct 2011 11:48:00 GMT

in 1921 Alfred Korzybski published Manhood of Humanity. in this work (and in subsequent editions) Korzybski lays out his basic premise that what makes humans unique is their ability to adapt and evolve more effectively than any other animal to date. he claims this is due to our ability to use "time" as a key abstraction; to evolve over time. the technique we use to accomplish this (he claims) is the ability to pass our knowledge down the generations (writing, etc.). we make "time" an ally in our evolution and adaptation. he called this uniquely human capability "time-binding."

effectively evolving over time; time-binding...

hmmm....

"There are two ways to slide easily through life; to believe everything or to doubt everything. Both ways save us from thinking. "

Alfred Korzybski, "Manhood of Humanity", 1921

binding examples for dist-net apps

when we write dist-net apps today, the problem of "binding" the client to the server must be addressed. i've mentioned this stuff before in this blog but i'll give a quick summary again since it's will fill some space here and enforce my selected POV for today[grin].

RPC-style: the client and server share the list of remote procedures including the names of the procedures and the calling details (arg list, types, etc.) and the expected return values/objects. clients are bound to the procedure list.
OO-style: the client and server share knowledge of the object graph including the hierarchy, member details, methods that can/should be applied to the objects, and the method of serializing and deserializing those objects. clients are bound to the object graph.
URI-style: the client and server share knowledge of the URI space (segments, parameters, etc.) and the payloads associated w/ those URIs. usually clients are given a set of rules on how to construct URIs and how to craft payloads to match. clients are bound to the URIs/rules.

these should all look/sound familiar; we deal with this stuff everyday. these are all examples of how developers attempt to animate a problem domain over a distributed network; how we write apps that get things done. basically, how devs work out a way to share understanding of the problem domain between client and server.

the evolvability problem

a key point here is to understand how these well-known models behave over time; how they handle "evolvability." for the most part, the styles identified above do not handle it well. for example, changes in the list of remote procedures, the shared object graphs, and/or the URI patterns all run the risk of (at least) freezing the exisiting client at some "static version" of the dist-net app. at worst, existing clients will "break" or become useless.

the common solution is to simply abandon old clients by asserting a new "version" of the app that handles the problem domain. this can be done by re-writing the client code and shipping that new code to everyone (Javascript anyone?), changing the URI pattern to start a "new" URI space that "old" clients never "see", or just making the changes and instructing all client devs to go back and re-write their code to match the new domain details.

IOW, there is no "evolving" here; each "version" is a "genetic dead end" so to speak. each major "modification" is a "greenfield project" for client coding.

but it does not need to be this way. it is possible to use the lessons from Korzybksi's "time-binding" model to craft dist-net apps that can sucessfully evolve over time. that can adapt to changes in the problem domain w/o needed to be re-coded or "left behind" as the app space moves on.

we need a new binding model

what some readers may have noticed is that, in the previous style examples the target of binding clients and servers is always a "first-class" aspect of the problem domain (the procedures, the objects, the network identifiers). it only stands to reason that changes in the problem domain will alter these collections and that alteration will run the risk of adversely affecting the over-all effectiveness of the participants (clients and servers) thereby reducing it's evolability.

one way to simply avoid this problem is to stop binding to "first-class" problem domain elements. instead, what is needed is a binding model that can support shared understanding between client and server, can effectively "animate" the problem domain over the dist-net, and still support change over time.

hypermedia binding

a solution to the evolving-over-time problem for dist-net apps is to bind clients and servers via a shared understanding of hypermedia controls. IOW, change the binding from first-class domain elements to an abstract expression of those elements; to elements (links and forms) that make "animating" dist-net apps possible.

let's add one more style to the binding list above...

Hypermedia-style: the client and server share understanding of the links and forms that may appear within a message. they agree on which hypermedia element represents a "safe" operation, which are "idemponent" or "non-idempotent", which elements support optional arguments (forms), which require payloads, and which support generalized rules about URI construction (URI templates, etc.). clients are bound to the hypermedia controls in a representation

in the above style, changes in the problem domain (new procedures, new objects, new URIs) do not "break" existing clients; they can continue to function and will instantly support the evolved problem domain. why? because the "binding" is orthagonal to the problem domain. the "binding" is to the hypermedia controls themselves; not what they "represent" at any point in time. whenever a form or link appears in a response, the client will be able to recognize, parse, and render it as needed. the system can successfully evolve over time.

the only risk of "breakage" is when the details of the hypmedia controls are changed. for example, new hypermedia elements appear (they will be ignored) or aspects of an existing hypermedia element are changed (this can "break" the client). but these changes are less likely to occur than changes in the problem domain and there are facilities for dealing with these changes, too (i.e. new media types, new transfer protocols, etc.).

bottom-line?

a hypermedia-oriented approach shifts the binding away from the volatile problem domain elements and toward the more stable hypermedia elements. in doing so, the evolvability of the system is increased. hypermedia-binding makes it easier for dist-net apps to evolve and adapt over time without running into a "dead-end"; without the need to declare a "version" for each meaningful modification in the system.

hypermedia-binding works at internet scale

hypermedia affordances

Tue, 27 Sep 2011 09:48:00 GMT

recently, i started thinking more about the concept of "affordances" and how that relates to my experiments in hypermedia designs. the trigger for these recent thought was a post by @danja and a few subsequent page entries to the W3C Wiki on RDF affordances. Danny's wiki entries point to a handful of other folks who have wrtten about affordances recently, too. all good stuff.

affordances

it's a funny word; made up word, actually:

The affordances of the environment are what it offers ... what it provides or furnishes, either for good or ill. The verb 'to afford' is found in the dictionary, but the noun 'affordance' is not. I have made it up (page 126).

The Ecological Approach to Visual Perception (Gibson, 1986)

not long after Gibson made up the word, Donald Norman used it, too:

...the term affordance refers to the perceived and actual properties of the thing, primarily those fundamental properties that determine just how the thing could possibly be used. (pg 9)

The [Design|Psychology] of Everyday Things (Norman, 1988)

years later, when trying to clarify what he meant by "hypermedia", Roy Fielding offered this:

When I say Hypertext, I mean the simultaneous presentation of information and controls such that the information becomes the affordance through which the user obtains choices and selects actions (slide #50).

Slide Presentation on REST (Fielding, 2008)

hypermedia affordance

if i follow the Gibson and Norman POV, i come to the point where "hypermedia affordances" should be recognizable "things" in the "landscape" that offer "actions." and these actions should be (IMO) related to hypermedia itself. so what would be the characteristics of hypermedia affordances? what would they "offer"? and how would they offer it?

right now i am of the opinion that a single set of aspects of hypermedia affordances can be identified. that this set would be universal to all hypermedia designs; no matter the protocol used (HTTP, FTP, etc.), the data format used (HTML, JSON, etc.). and that set of aspects should be enough to "construct" any hypermedia control needed in order to support a particular action in a distributed application.

the meaning and use of these affordance aspects must be shared by both client and server in a distributed network. IOW, when a server crafts a message that contains hypermedia affordances, the client has the same understanding of the meaning of that affordance as the server. to make that work, both parties need to know how to interpret the affordance. that's where the aspects come in.

i claim there is a stable set of aspects that can be found expressed in every affordance. and i claim that the set has only four members.

hypermedia affordance aspects

so here goes; my list of universal hypermedia affordance aspects:

Safety: the affordance offers either a safe action (HTML.A) or an unsafe action (ATOM.LINK@rel="edit").
Idempotence: the affordance represents either an idempotent action (HTML.FORM@method="get") or a non-idempotent action (HTML.FORM@method="post").
Mutability: the affordance is meant to support modification (mutable) by the client (HTML.FORM) or the affrodance is immutable (HTML.LINK).
Presentation: the result of activating the affordance should either be treated as a navigation (HTML.A) or as a transclusion (HTML.IMG).

that's it. these four aspects appear in every hypermedia control and can be used to create any hypermedia control needed in a dist. net. app.

examples

here are some examples:

HTML.IMG [safe, idempotent, immutable, transclusion]
HTML.FORM@method="post" [unsafe, non-idempotent, mutable, navigation]
HTML.A [safe, idempotent, immutable, navigation]
HTML.LINK@rel="stylesheet" [safe, idempotent, immutable, transclusion]
HTML.LINK@rel="rss" [safe, idempotent, immutable, navigation]

and the list goes on. this, i claim, works for any existing hypermedia type (Atom, SVG, CSS, etc.). i have been experimenting and i think, in some cases, the "aspect values" of some hypermedia controls are a bit vague (i.e. ATOM.LINK@rel="edit" = [unsafe, idempotent, mutable, navigation?]). right now i am of the opinion that this shows a vaguary in the control and/or documentation of that control, not a shortcoming in my set of aspects.

so...

so why is this interesting?

if the general theory holds true (that there is an invariant set of aspects shared by all hypermedia affordances, regardless of data format or protocol), then it should be possible to express the same hypermedia control in any format, for any protocol. IOW, code can be wrttten that "understands" the aspects, "understands" each hypermedia control, and can "act" accordingly.

it should be possible to create a matrix of controls and aspects. it should be possible to create a single client application (agent) that can properly recognize and parse any hypermedia type as long as the agent has access to the list of affordances and their aspects for that hypermedia type.

general theory of hypermedia?

universal hypermedia translator?

one format to rule them all?

three levels of abstraction

Thu, 25 Aug 2011 11:33:00 GMT

things seem to appear to me in threes. not sure why that is. usually because i almost always assume at least three possible solutions for any identified problem. an old habit.

the pledge

but there you are. i think in threes. and i've been thinking about this quite a bit lately. i've been inspired by several events in the last few weeks (three weeks maybe[grin]?). the most notable of these is last weekend's RESTFest 2011 event in Greenville, SC. we spent q good portion of the weekend hacking hypermedia using my ALPS Microblog experiment as a starting point. i learned several cool things. i think some other folks did, too.

the turn

one of the things that i got out of the weekend was a clarified understanding of the co-operative roles that play out in a hypermedia system when it is implemented over a distributed network. in our case, we were using the Web, HTTP, XHTML, and some specs for a microblog problem domain. the task was to build servers and clients that implement the spec and work both indepedently and in concert. IOW, loosely-coupled hypermedia clients and server, built independently, that achieve high interoperability.

believe me - it was cool!

along the way i discovered my experimental spec was lacking in some important ways. i won't go into the details here except to say that, in the end, i'd made the mistake of conflating app-level semantics w/ message-level semantics. the result of this kind of (common) mistake is often reduced interoperability. lucky for me, i had several very smart folks helping me find these problems and minimize their impact.

so smart, BTW, that i'm pretty sure some of them will end up w/ their names on the revised ALPS spec soon.

the prestige

see, my problem was mixing levels of abstraction in my implementation of the spec. but i get a bit ahead. first, the three levels of abstraction that have occupied my mind of late:

protocol (HTTP, FTP, etc.)
message (HTML, Atom, etc.)
domain (microblogging, accounting, etc.)

i've spent the last year or so focusing on identifying the relationship between the protocol abstraction and the message abstraction. i've done this by analyzing various existing media types; developing a set of H-Factors common to all message formats that want to express protocol-level features via hypermedia controls.

the next logical step is to analyze existing hypermedia implementations to locate and identify the general principles for expressing domain information within a hypermedia message. that's what the ALPS experiment is about. my problem was i mixed message semantics (layout and relationship of markup elements) with domain semantics (what data is required and how it should appear). it took just a couple client implementations right there on site to point out my short-comings. and i was most grateful.

because that's how i learned about my errors in abstraction for this particular set of circumstances.

i have a pretty good feel for the protocol-message interactions. IOW, i feel confident that i can suffciently express any protocol details within a message design (i.e. a MIME Media Type). now it's time to ramp up my thinking on the message-domain interactions. once that happens, i'll be looking to express the domain in a way that works for any message format. and that will pave the way for a clear abstraction of protocol, message, and domain. and that will lead, i think, to much better analysis and tooling; eventually to better developer experience during implementation, too.

believe me - it will be cool!