Re: Proposal for work on an efficient, browser-friendly, HTTP-based communication protocol for fine-grained information exchange

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for work on an efficient, browser-friendly, HTTP-based communication protocol for fine-grained information exchange

Wenbo Zhu-3
Very plausible .. and comments inline. - Wenbo
 
-------- Original Message --------
Subject: Proposal for work on an efficient, browser-friendly, HTTP-based communication protocol for fine-grained information exchange
Date: Thu, 12 Aug 2010 10:36:59 +0200
From: Julian Reschke <[hidden email]>
To: WebDAV <[hidden email]>

Proposal for work on an efficient, browser-friendly, HTTP-based
communication protocol for fine-grained information exchange

HTTP/1.1 (RFC 2616) already contains a set of tools for modifying
resources , namely the methods PUT, POST, and DELETE.

Many systems have been built on top of this, most of them in an ad-hoc
manner (which is ok when client and server are controlled by the same
developers).

We would like to cover some of the following use cases extending the
resource oriented model.

(1) An simple javascript based browser application should be able to
read fine-grained information (comparable to WebDAV properties) in a
simple manner using a defined JSON format to be consumed in an intuitive
fashion.

(2) A simple HTML Form should be able to write information in a patch
oriented manner containing both binary (file) data and fine-grained,
typed information using a multipart POST.

(3) A simple javascript application should be able to write information
in a patch oriented fashion using a defined JSON-diff PATCH content-type
to update fine-grained information.

There are also several extensions/applications of HTTP in this space,
such as:

- WebDAV (RFC 4918), which defines (a) a collection model and methods to
manipulate collections/namespaces, (b) a metadata (=property) model, and
(c) locking. Other RFCs add extensions on top of this, such as
Versioning (RFC 3253) and ACLs (RFC 3744).

- The Atom feed format (RFC 4287) and AtomPub (RFC 5023) use a simpler,
not necessarily hierarchic collection model (which, depending on the use
case, may be a plus), but does not provide many features WebDAV +
friends define. Notably, namespace operations are absent.

WebDAV and AtomPub have been very successful so far. WebDAV gets used
both as a plain remote filesystem protocol (as observed by clients being
shipped with all operating systems, and both Apache httpd and IIS
supporting it), and for specific applications, such as Versioning
(subversion), Calendaring (CalDAV), etc. The same is true for AtomPub,
which actually may not be used a lot in practice for the original use
case (feed authoring), but for many other things instead.

Both of those protocol specifications are not easily consumed by
websites and applications running current browsers and require a lot of
client-sided scripting to cover simple read and write use cases.

There's a proposal for a protocol called "JSOP", which addresses these
use cases, which we may want to consider as input for this work:
<http://www.slideshare.net/uncled/jsop>

So what's wrong with WebDAV?

Since the time WebDAV was designed, we have learned a lot how to use the
Web and HTTP. Such as:

- if you want to expose data for read operations, make it available to
GET, and assign URIs,

- consider cacheability, atomicity, and performance of sync operations
(for instance, syncing large collections),

- be careful with new HTTP methods -- avoid them for things that are not
of generic use (good: MKCOL, bad: MKCALENDAR) and keep in mind that
certain platforms (HTML forms, Flash...) can't use them,

- when defining formats, also define internet media types.

Also, in the last few months, new (and not so new) techniques have
finally been published as RFCs, such as:

- HTTP PATCH method (RFC 5789)

- HTTP Link Header and Link Relations Registry
(http://tools.ietf.org/html/draft-nottingham-http-link-header-10, in the
RFC Editor queue)

- Service Discovery through well-known URIs (RFC 5785)

Another potential building block are URI templates (work in progress:
http://tools.ietf.org/html/draft-gregorio-uritemplate-04)

Considering all of these pieces, it's quite obvious that there's a
number of specs that would be useful on their own, but could also,
combined together, form the basis of an interesting authoring protocol:


# Data Model

1) Define a collection model (hierarchy, naming), and a representation
format.

I have seen many debates around representation formats when the underlying meta-model is often ignored ... and the meta-model should cover, in addition to hierarchy, relations. And naming should allow for different representations too, e.g. with the URI template[] being one of them.
 
Can we re-use the WebDAV collection model here? Web application authors
probably would prefer a JSON representation, so can we simply define
this as an alternate representation of a DAV:multistatus description of
a collection?

2) Define namespace operations in terms of manipulating collection
representations (also consider a mapping to COPY/MOVE).

3) Define a media type to use with PATCH for modifying these
representations.

4) Define a property model (something like the intersection between
WebDAV properties and Java Content Repository (JSR-283) properties?)


# Authoring through HTML forms and POST

Define how POST with multipart/form-data (RFC 2388) can be used for
authoring both content and properties.


# URIs for collection browsing

Assign either hardwired or discoverable URIs for inspecting collections
(URI templates?). Or maybe link relations for collection navigation
(similar work for versioning: RFC 5829).


# Improvements to WebDAV

1) Clarify how MOVE and COPY can operate on non-WebDAV resources (this
question comes up quite frequently).

2) Define how to use POST on WebDAV collections to add members (done:
see http://greenbytes.de/tech/webdav/#draft-reschke-webdav-post, in RFC
Editor queue).

3) Define media types (multiple?) for DAV:multistatus.

4) Define a discovery mechanism for GETtable representations of
PROPFIND/REPORT results (old proposal:
http://greenbytes.de/tech/webdav/draft-reschke-http-get-location-latest.html).

5) Define a mapping between link-typed WebDAV properties and generic
Link relations (see proposal:
http://lists.w3.org/Archives/Public/w3c-dist-auth/2008OctDec/0026.html).

Although some of this will only be partially related to WebDAV, we think
that this mailing list might be a good venue for discussion.


Expected deliverables from this activity would be:

1) Definition of a very simply data model and a representation format
for it (required JSON, optionally XML).

2) A format suitable for manipulating the data format above using PATCH
(potentially tunneled through POST).

3) A binding from multipart/form-data/POST to this model.

4) A separate (?) document explaining how these ingredients would be
combined in practice.

Extensions to WebDAV and mappings from/to WebDAV could be useful, but
would not be a core part of this activity. (That is, we can do without
if no volunteers speak up).

Resource-based concurrency-control and sync (revision logs) specs may be developed on top of these deliverables as well.
 

Note  that not all of these specs necessarily need to be on the
standards  track; for instance, there might be candidates for
Informational RFCs as  well (see
<http://tools.ietf.org/html/rfc2026#section-4> for details).


Feedback appreciated.

Julian Reschke
David Nüscheler



PS: people not familiar with the IETF may want to have a look at
<http://www.ietf.org/tao.html>



Reply | Threaded
Open this post in threaded view
|

Re: Proposal for work on an efficient, browser-friendly, HTTP-based communication protocol for fine-grained information exchange

Wenbo Zhu-3
[re-repost]

Very plausible .. and comments inline. - Wenbo
 
-------- Original Message --------

Subject: Proposal for work on an efficient, browser-friendly, HTTP-based communication protocol for fine-grained information exchange
Date: Thu, 12 Aug 2010 10:36:59 +0200
From: Julian Reschke <[hidden email]>
To: WebDAV <[hidden email]>

Proposal for work on an efficient, browser-friendly, HTTP-based
communication protocol for fine-grained information exchange

HTTP/1.1 (RFC 2616) already contains a set of tools for modifying
resources , namely the methods PUT, POST, and DELETE.

Many systems have been built on top of this, most of them in an ad-hoc
manner (which is ok when client and server are controlled by the same
developers).

We would like to cover some of the following use cases extending the
resource oriented model.

(1) An simple javascript based browser application should be able to
read fine-grained information (comparable to WebDAV properties) in a
simple manner using a defined JSON format to be consumed in an intuitive
fashion.

(2) A simple HTML Form should be able to write information in a patch
oriented manner containing both binary (file) data and fine-grained,
typed information using a multipart POST.

(3) A simple javascript application should be able to write information
in a patch oriented fashion using a defined JSON-diff PATCH content-type
to update fine-grained information.

There are also several extensions/applications of HTTP in this space,
such as:

- WebDAV (RFC 4918), which defines (a) a collection model and methods to
manipulate collections/namespaces, (b) a metadata (=property) model, and
(c) locking. Other RFCs add extensions on top of this, such as
Versioning (RFC 3253) and ACLs (RFC 3744).

- The Atom feed format (RFC 4287) and AtomPub (RFC 5023) use a simpler,
not necessarily hierarchic collection model (which, depending on the use
case, may be a plus), but does not provide many features WebDAV +
friends define. Notably, namespace operations are absent.

WebDAV and AtomPub have been very successful so far. WebDAV gets used
both as a plain remote filesystem protocol (as observed by clients being
shipped with all operating systems, and both Apache httpd and IIS
supporting it), and for specific applications, such as Versioning
(subversion), Calendaring (CalDAV), etc. The same is true for AtomPub,
which actually may not be used a lot in practice for the original use
case (feed authoring), but for many other things instead.

Both of those protocol specifications are not easily consumed by
websites and applications running current browsers and require a lot of
client-sided scripting to cover simple read and write use cases.

There's a proposal for a protocol called "JSOP", which addresses these
use cases, which we may want to consider as input for this work:
<http://www.slideshare.net/uncled/jsop>

So what's wrong with WebDAV?

Since the time WebDAV was designed, we have learned a lot how to use the
Web and HTTP. Such as:

- if you want to expose data for read operations, make it available to
GET, and assign URIs,

- consider cacheability, atomicity, and performance of sync operations
(for instance, syncing large collections),

- be careful with new HTTP methods -- avoid them for things that are not
of generic use (good: MKCOL, bad: MKCALENDAR) and keep in mind that
certain platforms (HTML forms, Flash...) can't use them,

- when defining formats, also define internet media types.

Also, in the last few months, new (and not so new) techniques have
finally been published as RFCs, such as:

- HTTP PATCH method (RFC 5789)

- HTTP Link Header and Link Relations Registry
(http://tools.ietf.org/html/draft-nottingham-http-link-header-10, in the
RFC Editor queue)

- Service Discovery through well-known URIs (RFC 5785)

Another potential building block are URI templates (work in progress:
http://tools.ietf.org/html/draft-gregorio-uritemplate-04)

Considering all of these pieces, it's quite obvious that there's a
number of specs that would be useful on their own, but could also,
combined together, form the basis of an interesting authoring protocol:


# Data Model

1) Define a collection model (hierarchy, naming), and a representation
format.

I have seen many debates around representation formats when the underlying meta-model is often ignored ... and the meta-model should cover, in addition to hierarchy, relations. And naming should allow for different representations too, e..g. with the URI template[] being one of them.
 
Can we re-use the WebDAV collection model here? Web application authors

probably would prefer a JSON representation, so can we simply define
this as an alternate representation of a DAV:multistatus description of
a collection?

2) Define namespace operations in terms of manipulating collection
representations (also consider a mapping to COPY/MOVE).

3) Define a media type to use with PATCH for modifying these
representations.

4) Define a property model (something like the intersection between
WebDAV properties and Java Content Repository (JSR-283) properties?)


# Authoring through HTML forms and POST

Define how POST with multipart/form-data (RFC 2388) can be used for
authoring both content and properties.


# URIs for collection browsing

Assign either hardwired or discoverable URIs for inspecting collections
(URI templates?). Or maybe link relations for collection navigation
(similar work for versioning: RFC 5829).


# Improvements to WebDAV

1) Clarify how MOVE and COPY can operate on non-WebDAV resources (this
question comes up quite frequently).

2) Define how to use POST on WebDAV collections to add members (done:
see http://greenbytes.de/tech/webdav/#draft-reschke-webdav-post, in RFC
Editor queue).

3) Define media types (multiple?) for DAV:multistatus.

4) Define a discovery mechanism for GETtable representations of
PROPFIND/REPORT results (old proposal:
http://greenbytes.de/tech/webdav/draft-reschke-http-get-location-latest.html).

5) Define a mapping between link-typed WebDAV properties and generic
Link relations (see proposal:
http://lists.w3.org/Archives/Public/w3c-dist-auth/2008OctDec/0026.html).

Although some of this will only be partially related to WebDAV, we think
that this mailing list might be a good venue for discussion.


Expected deliverables from this activity would be:

1) Definition of a very simply data model and a representation format
for it (required JSON, optionally XML).

2) A format suitable for manipulating the data format above using PATCH
(potentially tunneled through POST).

3) A binding from multipart/form-data/POST to this model.

4) A separate (?) document explaining how these ingredients would be
combined in practice.

Extensions to WebDAV and mappings from/to WebDAV could be useful, but
would not be a core part of this activity. (That is, we can do without
if no volunteers speak up).

Resource-based concurrency-control and sync (revision logs) specs may be developed on top of these deliverables as well.
 

Note  that not all of these specs necessarily need to be on the
standards  track; for instance, there might be candidates for
Informational RFCs as  well (see
<http://tools.ietf.org/html/rfc2026#section-4> for details).


Feedback appreciated.

Julian Reschke
David Nüscheler



PS: people not familiar with the IETF may want to have a look at
<http://www.ietf.org/tao.html>




Reply | Threaded
Open this post in threaded view
|

Re: Proposal for work on an efficient, browser-friendly, HTTP-based communication protocol for fine-grained information exchange

Julian Reschke
In reply to this post by Wenbo Zhu-3
On 13.08.2010 10:02, Wenbo Zhu wrote:

> ...
>     # Data Model
>
>     1) Define a collection model (hierarchy, naming), and a representation
>     format.
>
> I have seen many debates around representation formats when the
> underlying meta-model is often ignored ... and the meta-model should
> cover, in addition to hierarchy, relations. And naming should allow for
> different representations too, e.g. with the URI template[] being one of
> them.

I think we need to be careful; if we over-engineer the data model we may
scare away parts of the audience we want to include. It should be
possible to define something that has an easy-to-use JSON representation
but still have a solid model in the background.

Including relations into the model makes sense to me. Links and link
relations are important.

I'm not sure I understand the point about naming; could you elaborate?

> ...
>     Expected deliverables from this activity would be:
>
>     1) Definition of a very simply data model and a representation format
>     for it (required JSON, optionally XML).
>
>     2) A format suitable for manipulating the data format above using PATCH
>     (potentially tunneled through POST).
>
>     3) A binding from multipart/form-data/POST to this model.
>
>     4) A separate (?) document explaining how these ingredients would be
>     combined in practice.
>
>     Extensions to WebDAV and mappings from/to WebDAV could be useful, but
>     would not be a core part of this activity. (That is, we can do without
>     if no volunteers speak up).
>
>
> Resource-based concurrency-control and sync (revision logs) specs may be
> developed on top of these deliverables as well.

Concurrency control as in locking? (be it optimistic or pessimistic)

Best regards, Julian

Reply | Threaded
Open this post in threaded view
|

Re: Proposal for work on an efficient, browser-friendly, HTTP-based communication protocol for fine-grained information exchange

Wenbo Zhu-3


On Sun, Aug 15, 2010 at 5:10 AM, Julian Reschke <[hidden email]> wrote:
On 13.08.2010 10:02, Wenbo Zhu wrote:
...

   # Data Model

   1) Define a collection model (hierarchy, naming), and a representation
   format.

I have seen many debates around representation formats when the
underlying meta-model is often ignored ... and the meta-model should
cover, in addition to hierarchy, relations. And naming should allow for
different representations too, e.g. with the URI template[] being one of
them.

I think we need to be careful; if we over-engineer the data model we may scare away parts of the audience we want to include. It should be possible to define something that has an easy-to-use JSON representation but still have a solid model in the background.

Including relations into the model makes sense to me. Links and link relations are important.

I'm not sure I understand the point about naming; could you elaborate?
Separate the logic naming scheme from its syntactic representation ...  and the former should be derived from the "background model". When relationships are included, entities may be addressed via links too.

 

...

   Expected deliverables from this activity would be:

   1) Definition of a very simply data model and a representation format
   for it (required JSON, optionally XML).

   2) A format suitable for manipulating the data format above using PATCH
   (potentially tunneled through POST).

   3) A binding from multipart/form-data/POST to this model.

   4) A separate (?) document explaining how these ingredients would be
   combined in practice.

   Extensions to WebDAV and mappings from/to WebDAV could be useful, but
   would not be a core part of this activity. (That is, we can do without
   if no volunteers speak up).


Resource-based concurrency-control and sync (revision logs) specs may be
developed on top of these deliverables as well.

Concurrency control as in locking? (be it optimistic or pessimistic)
Yes.
 

Best regards, Julian

Reply | Threaded
Open this post in threaded view
|

Re: Proposal for work on an efficient, browser-friendly, HTTP-based communication protocol for fine-grained information exchange

Julian Reschke
On 19.08.2010 20:00, Wenbo Zhu wrote:

>     I think we need to be careful; if we over-engineer the data model we
>     may scare away parts of the audience we want to include. It should
>     be possible to define something that has an easy-to-use JSON
>     representation but still have a solid model in the background.
>
>     Including relations into the model makes sense to me. Links and link
>     relations are important.
>
>     I'm not sure I understand the point about naming; could you elaborate?
>
> Separate the logic naming scheme from its syntactic representation ...
>   and the former should be derived from the "background model". When

Yes. We just need to make sure that by doing it this way, the wire
format doesn't get more complicated than necessary.

> relationships are included, entities may be addressed via links too.
>            Extensions to WebDAV and mappings from/to WebDAV could be
>         useful, but
>             would not be a core part of this activity. (That is, we can
>         do without
>             if no volunteers speak up).
>
>
>         Resource-based concurrency-control and sync (revision logs)
>         specs may be
>         developed on top of these deliverables as well.
>
>
>     Concurrency control as in locking? (be it optimistic or pessimistic)
>
> Yes.

I think if we do things right, optimistic locking based on HTTP/1.1
conditional requests and etags will work.

I'm not convinced there's big interest for pessimistic locking (à la
WebDAV locking) at this point of time.

Best regards, Julian

Reply | Threaded
Open this post in threaded view
|

Re: Proposal for work on an efficient, browser-friendly, HTTP-based communication protocol for fine-grained information exchange

David Nuescheler-3
Hi All,

Julian and I (and occasionally many others) have been discussing a
development effort like this for quite while.
I was involved in mapping the scope of WebDAV (and friends) into a
Java API called JCR[1][2][3] (on which for example Geoff, Julian and
Roy participated) and due to my Day[4] job (pun intended) I find
myself very often in a situation where I need have a standard
web-browser or client-sided web-application interact with a server to
exchange fine-grained information.

Thanks a lot to everybody for all the comments and input.

I guess my main take from this is that I completely agree that we need
to separate the "model"-conversation from the
"format/binding"-conversation.

I would like to mention though that in my mind the goal of making this
effort relevant, efficient and simple is extremely important as well.
While I appreciate that the separation of the discussions, I would
like to volunteer to work on bindings to a JSON + PATCH (multipart
POST) very early on in the process and keep it in sync with the model
as a living set of examples for the interaction with the more abstract
model, to ensure that we keep things practical.

I think there are of a large number of very similar, JSON/POST-based
"protocols" that are defined in an ad-hoc manner by developers. So
there should be quite a bit of experience out there, that can help us
gauge the importance some of requirements fairly quickly. I think if
we manage to take the combined the experience from WebDAV, AtomPub and
JCR (and possibly more domain specific efforts like CMIS) in terms of
the overall scope and the initial relevance of certain features we
should be in great shape.

regards,
david

[1] http://jcp.org/en/jsr/detail?id=170
[2] http://jcp.org/en/jsr/detail?id=283
[3] http://jcp.org/en/jsr/detail?id=333
[4] http://www.day.com

--
David Nuescheler
Chief Technology Officer
mailto: [hidden email]

web:  http://www.day.com/ http://dev.day.com
twitter: @davidnuescheler


Reply | Threaded
Open this post in threaded view
|

Re: Proposal for work on an efficient, browser-friendly, HTTP-based communication protocol for fine-grained information exchange

Michael Wechner
David Nuescheler wrote:

> Hi All,
>
> Julian and I (and occasionally many others) have been discussing a
> development effort like this for quite while.
> I was involved in mapping the scope of WebDAV (and friends) into a
> Java API called JCR[1][2][3] (on which for example Geoff, Julian and
> Roy participated) and due to my Day[4] job (pun intended) I find
> myself very often in a situation where I need have a standard
> web-browser or client-sided web-application interact with a server to
> exchange fine-grained information.
>  

can you give a "real-world" example such that I can understand better :-) ?

And also where the existing "protocols" are lacking?

Thanks

Michael

> Thanks a lot to everybody for all the comments and input.
>
> I guess my main take from this is that I completely agree that we need
> to separate the "model"-conversation from the
> "format/binding"-conversation.
>
> I would like to mention though that in my mind the goal of making this
> effort relevant, efficient and simple is extremely important as well.
> While I appreciate that the separation of the discussions, I would
> like to volunteer to work on bindings to a JSON + PATCH (multipart
> POST) very early on in the process and keep it in sync with the model
> as a living set of examples for the interaction with the more abstract
> model, to ensure that we keep things practical.
>
> I think there are of a large number of very similar, JSON/POST-based
> "protocols" that are defined in an ad-hoc manner by developers. So
> there should be quite a bit of experience out there, that can help us
> gauge the importance some of requirements fairly quickly. I think if
> we manage to take the combined the experience from WebDAV, AtomPub and
> JCR (and possibly more domain specific efforts like CMIS) in terms of
> the overall scope and the initial relevance of certain features we
> should be in great shape.
>
> regards,
> david
>
> [1] http://jcp.org/en/jsr/detail?id=170
> [2] http://jcp.org/en/jsr/detail?id=283
> [3] http://jcp.org/en/jsr/detail?id=333
> [4] http://www.day.com
>
>  



Reply | Threaded
Open this post in threaded view
|

Data Model

Julian Reschke
In reply to this post by David Nuescheler-3
On 23.08.2010 15:58, David Nuescheler wrote:
> ...
> I guess my main take from this is that I completely agree that we need
> to separate the "model"-conversation from the
> "format/binding"-conversation.
> ...

Ok,

here's a list of things we need to consider (probably incomplete, feel
free to add)

1. Collections

1.1 Hierarchy: WebDAV collections are hierarchical. A "direct" member of
a collection has the parent collections URI + one additional path
segment. JCR same. This also means that relative paths do the obvious
thing. CMIS/AtomPub: no constraints on the paths (AFAIU).

1.2 Multiple containment: allowed in WebDAV through multiple bindings,
in JCR through shared nodes. CMIS/AtomPub: not constrained anyway.

1.3 Identification: WebDAV: the same of a resource within a collection
is a unique identifier. JCR: same (except that for same-name-siblings,
the array index may change when siblings are removed).

1.4 Ordering: optional features in WebDAV and JCR.

1.5 Member naming: in WebDAV by path segment (URI syntax), in JCR per
optional namespace name + path segment.

1.6 Name encoding: in WebDAV per spec ASCII (+ URI percent encoding), in
JCR Unicode.


2. Properties

2.1 Cardinality: properties can only occur once on a resource, but there
are ways to express lists (WebDAV, JCR)

2.2 Typing: optional in WebDAV (see RFC 4316). Set of predefined types
in JCR (int, float, string, date, URI, ...)

2.3 Naming: namespace name + local name (WebDAV, JCR)


3. Content

3.1 A single binary stream per HTTP in WebDAV (ignoring content
negotiation for now which is tricky in authoring); modeled as binary
property in JCR (with only conventions on naming)


Best regards, Julian

Reply | Threaded
Open this post in threaded view
|

Re: Proposal for work on an efficient, browser-friendly, HTTP-based communication protocol for fine-grained information exchange

David Nuescheler-3
In reply to this post by Michael Wechner
Hi Michael,

Thanks a lot for your question.

On Thu, Aug 26, 2010 at 10:52 AM, Michael Wechner
<[hidden email]> wrote:
> can you give a "real-world" example such that I can understand better :-) ?
> And also where the existing "protocols" are lacking?

I am more than happy to give you a couple of real-life use cases that
may give you a little bit of insight on what i have in mind.

Let's assume as a starting point that I have a standard HTML form in a
browser that I want a user to enter information. While it doesn't
really matter what the application is, to make it really tangible
let's say it is a form that I want to let a user update their profile
information, such as their first name, last name and birthdate.

If we assume that this is fine-grained information that is persisted
on the server, one would assume that the user profile could be
considered a resource (in the WebDAV sense) and the fine-grained
information (first name, last name and birthdate) would lend itself to
be properties on that resource (in the webdav sense).

Assuming that I wanted to transport this information to a WebDAV
server using WebDAV as the communication protocol between my browser
and my WebDAV server I would find myself in a situation where I would
have to write a very healthy chunk of javascript code working with XHR
to come up with the PROPPATCH method and the respective XML for the
body of the request to be able to persist my information to WebDAV
server.
If we extend the use case to a user uploading their profile picture,
then the the task to interact with my WebDAV server goes from
"difficult" to "impossible" using my standard browser.

The goal of this effort is to define an interaction pattern that
allows a standard browser to interact with the server in a
fine-grained and efficient manner, yet honoring a lot of the semantics
and model that have been established around WebDAV.

So the above example, in my mind could result in a very simple form
that could look like this. In straight up (somewhat simplified) html
...

---
<form method="POST" action="/profiles/david">
        <input name="firstname" />
        <input name="lastname" />
        <input name="birthdate" />
        <input name="picture" type="file" />
        <input type="submit" />
</form>
---

...and would not require the hundreds of lines of code javascript and
a very cooperative browser to allow me to update my profile.

Further more if we assume that if the client-sided javascript
application becomes more complex and would like to interact with the
fine-grained information from a JavaScript/XHR standpoint there would
be the ability to easily access with a GET the above information as
something like:

---
{
firstname: "David",
lastname: "Nuescheler",
birthdate: "Oct 31 1974"
}
---

... or update the above with something along the lines of a PATCH or a
POST with a body similar to...

---
-birthdate
^lastModifiedBy : "me"
---

... which would remove the birthdate property and update the
lastModifiedBy, directly from a Javascript application.


As suggested in the thread before, I agree that the conversations
around the model that we are operating on and the respective bindings
to specific formats need to be separate conversations, but I think to
make it more tangible and real-life as you requested I thought that it
may make sense to wrap it into the example.

Please let me know if that addresses your question or if it is
something else that you were looking for.

regards,
david

On Thu, Aug 26, 2010 at 10:52 AM, Michael Wechner
<[hidden email]> wrote:

> David Nuescheler wrote:
>>
>> Hi All,
>>
>> Julian and I (and occasionally many others) have been discussing a
>> development effort like this for quite while.
>> I was involved in mapping the scope of WebDAV (and friends) into a
>> Java API called JCR[1][2][3] (on which for example Geoff, Julian and
>> Roy participated) and due to my Day[4] job (pun intended) I find
>> myself very often in a situation where I need have a standard
>> web-browser or client-sided web-application interact with a server to
>> exchange fine-grained information.
>>
>
> can you give a "real-world" example such that I can understand better :-) ?
>
> And also where the existing "protocols" are lacking?
>
> Thanks
>
> Michael
>>
>> Thanks a lot to everybody for all the comments and input.
>>
>> I guess my main take from this is that I completely agree that we need
>> to separate the "model"-conversation from the
>> "format/binding"-conversation.
>>
>> I would like to mention though that in my mind the goal of making this
>> effort relevant, efficient and simple is extremely important as well.
>> While I appreciate that the separation of the discussions, I would
>> like to volunteer to work on bindings to a JSON + PATCH (multipart
>> POST) very early on in the process and keep it in sync with the model
>> as a living set of examples for the interaction with the more abstract
>> model, to ensure that we keep things practical.
>>
>> I think there are of a large number of very similar, JSON/POST-based
>> "protocols" that are defined in an ad-hoc manner by developers. So
>> there should be quite a bit of experience out there, that can help us
>> gauge the importance some of requirements fairly quickly. I think if
>> we manage to take the combined the experience from WebDAV, AtomPub and
>> JCR (and possibly more domain specific efforts like CMIS) in terms of
>> the overall scope and the initial relevance of certain features we
>> should be in great shape.
>>
>> regards,
>> david
>>
>> [1] http://jcp.org/en/jsr/detail?id=170
>> [2] http://jcp.org/en/jsr/detail?id=283
>> [3] http://jcp.org/en/jsr/detail?id=333
>> [4] http://www.day.com
>>
>>
>
>



--
David Nuescheler
Chief Technology Officer
mailto: [hidden email]

web:  http://www.day.com/ http://dev.day.com
twitter: @davidnuescheler

Reply | Threaded
Open this post in threaded view
|

Re: Data Model

David Nuescheler-3
In reply to this post by Julian Reschke
Hi Julian,

thanks for kickstarting the conversation...

Here my initial take on the topics below.

> 1. Collections
> 1.1 Hierarchy: WebDAV collections are hierarchical. A "direct" member of a
> collection has the parent collections URI + one additional path segment. JCR
> same. This also means that relative paths do the obvious thing.
> CMIS/AtomPub: no constraints on the paths (AFAIU).
I think the WebDAV hierarchy approach is great and very important
since referencing by (relative) URL is core to basic web things.

> 1.2 Multiple containment: allowed in WebDAV through multiple bindings, in
> JCR through shared nodes. CMIS/AtomPub: not constrained anyway.
While I think that multiple containment is interesting, it is
definitely not at the top of my priority list.

> 1.3 Identification: WebDAV: the same of a resource within a collection is a
> unique identifier. JCR: same (except that for same-name-siblings, the array
> index may change when siblings are removed).
I think the WebDAV way works perfectly.

> 1.4 Ordering: optional features in WebDAV and JCR.
I think this is more important for fine-grained content since when we
talk about DOM-like structures that are persisted on the server of
course the sort order makes a big difference. (As a side comment:
mapping things to JSON in the back of my mind creates a bit of tension
here, since sort-order of JSON object is undefined, while all
implementations in browsers keep the sort-order, technically a JSON
parser is not obligated to do that...)

> 1.5 Member naming: in WebDAV by path segment (URI syntax), in JCR per
> optional namespace name + path segment.
I am not sure if we need a special (xml-like) namespace management, it
seems like a lot of overhead.

> 1.6 Name encoding: in WebDAV per spec ASCII (+ URI percent encoding), in JCR
> Unicode.
ASCII + URI encoding sounds great

> 2. Properties
> 2.1 Cardinality: properties can only occur once on a resource, but there are
> ways to express lists (WebDAV, JCR)
I think there is no need to change that.

> 2.2 Typing: optional in WebDAV (see RFC 4316). Set of predefined types in
> JCR (int, float, string, date, URI, ...)
If we look at the types in a browser javascript runtime it seems that
number, strings, boolean and Date seem like obvious candiates.

> 2.3 Naming: namespace name + local name (WebDAV, JCR)
Great.

> 3. Content
> 3.1 A single binary stream per HTTP in WebDAV (ignoring content negotiation
> for now which is tricky in authoring); modeled as binary property in JCR
> (with only conventions on naming)
I think in JCR we went all out and in my mind went too far with
binaries. I think I would be happy with having a single optional
binary stream.
More importantly though since this is about fine-grained information
the typical case will be having "no" binary at all, but just a tree of
properties (and "nodes?").

regards,
david

Reply | Threaded
Open this post in threaded view
|

Re: Data Model

Julian Reschke
On 29.08.2010 21:05, David Nuescheler wrote:
> Hi Julian,
>
> thanks for kickstarting the conversation...

:-)

I was planning to actually insert my opinion as well, but was waiting
for other's views first.

Here we go:

> Here my initial take on the topics below.
>
>> 1. Collections
>> 1.1 Hierarchy: WebDAV collections are hierarchical. A "direct" member of a
>> collection has the parent collections URI + one additional path segment. JCR
>> same. This also means that relative paths do the obvious thing.
>> CMIS/AtomPub: no constraints on the paths (AFAIU).
> I think the WebDAV hierarchy approach is great and very important
> since referencing by (relative) URL is core to basic web things.

Yes.

>> 1.2 Multiple containment: allowed in WebDAV through multiple bindings, in
>> JCR through shared nodes. CMIS/AtomPub: not constrained anyway.
> While I think that multiple containment is interesting, it is
> definitely not at the top of my priority list.

Indeed. As we have seen in WebDAV and JCR, it can be added later on
without affecting existing code (just be a bit careful when phrasing
things, not confusing the identifier with the thing being identified).

>> 1.3 Identification: WebDAV: the same of a resource within a collection is a
>> unique identifier. JCR: same (except that for same-name-siblings, the array
>> index may change when siblings are removed).
> I think the WebDAV way works perfectly.

+1

>> 1.4 Ordering: optional features in WebDAV and JCR.
> I think this is more important for fine-grained content since when we
> talk about DOM-like structures that are persisted on the server of
> course the sort order makes a big difference. (As a side comment:
> mapping things to JSON in the back of my mind creates a bit of tension
> here, since sort-order of JSON object is undefined, while all
> implementations in browsers keep the sort-order, technically a JSON
> parser is not obligated to do that...)
>
>> 1.5 Member naming: in WebDAV by path segment (URI syntax), in JCR per
>> optional namespace name + path segment.
> I am not sure if we need a special (xml-like) namespace management, it
> seems like a lot of overhead.
>
>> 1.6 Name encoding: in WebDAV per spec ASCII (+ URI percent encoding), in JCR
>> Unicode.
> ASCII + URI encoding sounds great

Yes, but I think we need to require that percent encoding actually is
used to encode UTF-8 sequences. In WebDAV, the interop problems between
*nix boxes (octet sequences as filenames) and Windows boxes (Unicode)
have been painful.

>> 2. Properties
>> 2.1 Cardinality: properties can only occur once on a resource, but there are
>> ways to express lists (WebDAV, JCR)
> I think there is no need to change that.
>
>> 2.2 Typing: optional in WebDAV (see RFC 4316). Set of predefined types in
>> JCR (int, float, string, date, URI, ...)
> If we look at the types in a browser javascript runtime it seems that
> number, strings, boolean and Date seem like obvious candiates.

Yes. I also believe that special casing links would be good.

>> 2.3 Naming: namespace name + local name (WebDAV, JCR)
> Great.
>
>> 3. Content
>> 3.1 A single binary stream per HTTP in WebDAV (ignoring content negotiation
>> for now which is tricky in authoring); modeled as binary property in JCR
>> (with only conventions on naming)
> I think in JCR we went all out and in my mind went too far with
> binaries. I think I would be happy with having a single optional
> binary stream.
> More importantly though since this is about fine-grained information
> the typical case will be having "no" binary at all, but just a tree of
> properties (and "nodes?").

Would a zero length content work as well?

Best regards, Julian

Reply | Threaded
Open this post in threaded view
|

Re: Data Model

David Nuescheler-3
Hi Julian,

thanks for the additional color.

Let me chime in on one aspect...

>> I think in JCR we went all out and in my mind went too far with
>> binaries. I think I would be happy with having a single optional
>> binary stream.
>> More importantly though since this is about fine-grained information
>> the typical case will be having "no" binary at all, but just a tree of
>> properties (and "nodes?").
> Would a zero length content work as well?

Well, personally, I would rather avoid that route.

I thought about this from various different aspects and while it of
course works from an implementation and usage standpoint I would argue
that it sets the wrong expectation and targets the wrong use cases.

In my mind the general case is that the "nodes" (or the lack of a
better term) do not have a "binary stream" associated, and in
exceptional cases they do. I see the fine-grained nature more similar
to rows of a table in relational database.
So in my mind it is important to identify the "binary content" as the
special case and make sure that the "binary content-less" concept is
treated as the general case, and not the other way around.

I realize that this is just a matter of setting the perception
correctly but that's precisely why would like to be careful ;)

regards,
david

Reply | Threaded
Open this post in threaded view
|

Re: Data Model

Julian Reschke
On 30.08.2010 20:08, David Nuescheler wrote:

> Hi Julian,
>
> thanks for the additional color.
>
> Let me chime in on one aspect...
>
>>> I think in JCR we went all out and in my mind went too far with
>>> binaries. I think I would be happy with having a single optional
>>> binary stream.
>>> More importantly though since this is about fine-grained information
>>> the typical case will be having "no" binary at all, but just a tree of
>>> properties (and "nodes?").
>> Would a zero length content work as well?
>
> Well, personally, I would rather avoid that route.
>
> I thought about this from various different aspects and while it of
> course works from an implementation and usage standpoint I would argue
> that it sets the wrong expectation and targets the wrong use cases.
>
> In my mind the general case is that the "nodes" (or the lack of a
> better term) do not have a "binary stream" associated, and in
> exceptional cases they do. I see the fine-grained nature more similar
> to rows of a table in relational database.
> So in my mind it is important to identify the "binary content" as the
> special case and make sure that the "binary content-less" concept is
> treated as the general case, and not the other way around.
>
> I realize that this is just a matter of setting the perception
> correctly but that's precisely why would like to be careful ;)
> ...

Understood.

I think once we have defined addressing we'll have to get back to this
question -- if a node gets a URI we'll have to answer what its GETtable
representation will be...

Best regards, Julian

Reply | Threaded
Open this post in threaded view
|

Re: Data Model

Julian Reschke
In reply to this post by David Nuescheler-3
On 29.08.2010 21:05, David Nuescheler wrote:

> ...
>> 1.4 Ordering: optional features in WebDAV and JCR.
> I think this is more important for fine-grained content since when we
> talk about DOM-like structures that are persisted on the server of
> course the sort order makes a big difference. (As a side comment:
> mapping things to JSON in the back of my mind creates a bit of tension
> here, since sort-order of JSON object is undefined, while all
> implementations in browsers keep the sort-order, technically a JSON
> parser is not obligated to do that...)
> ...

If ordering of collection members is relevant, then mapping to a
Javascript array (instead of object) is probably required:

    An object is an unordered collection of zero or more name/value
    pairs, where a name is a string and a value is a string, number,
    boolean, null, object, or array.

(<http://tools.ietf.org/html/rfc4627#section-1>)

Best regards, Julian

Reply | Threaded
Open this post in threaded view
|

Re: Data Model

David Nuescheler-3
> If ordering of collection members is relevant, then mapping to a Javascript
> array (instead of object) is probably required:
sigh... i know... unfortunately...

regards,
david

Reply | Threaded
Open this post in threaded view
|

Data Model, as seen from WebDAV

Julian Reschke
In reply to this post by Julian Reschke
Hi,

going back to my notes and David's feedback, we seem to be very close to
use a profile of the WebDAV data model:

> 1. Collections
>
> 1.1 Hierarchy: WebDAV collections are hierarchical. A "direct" member of a
> collection has the parent collection's URI + one additional path segment. JCR:
> same. This also means that relative paths do the obvious thing. CMIS/AtomPub:
> no constraints on the paths (AFAIU).

-> WebDAV model (hierarchical collection matching HTTP URI path syntax).

> 1.2 Multiple containment: allowed in WebDAV through multiple bindings, in JCR
> through shared nodes. CMIS/AtomPub: not constrained anyway.

Not now.

> 1.3 Identification: WebDAV: the name of a resource within a collection is a
> unique identifier. JCR: same (except that for same-name-siblings, the array
> index may change when siblings are removed/inserted).

-> WebDAV model (where index notation would be used if we want same-name
siblings)

> 1.4 Ordering: optional features in WebDAV and JCR.

-> WebDAV model (RFC 3648)

> 1.5 Member naming: in WebDAV by path segment (URI syntax), in JCR per optional
> namespace name + path segment.
>
> 1.6 Name encoding: in WebDAV per spec ASCII (+ URI percent encoding), in JCR
> Unicode. URI syntax requires defining the character encoding for interoperability.

-> WebDAV, with a constraint that percent-encoded stuff needs to be NFC
UTF-8; alternatively use Unicode (NFC?) and define the mapping to path
segment when mapping to a URI.

> 2. Properties
>
> 2.1 Cardinality: properties can only occur once on a resource, but there are
> ways to express lists (WebDAV, JCR)

-> WebDAV model

! This will make it tricky to map from/to RDF. Just saying.

> 2.2 Typing: optional in WebDAV (see RFC 4316). Set of predefined types in JCR
> (int, float, string, date, URI, ...)

-> WebDAV model, augmented with bits from RFC 4316, and restricting some
forms (such as mixed XML content)

Open question: do we want types not directly available in JSON, such as
links?

> 2.3 Naming: namespace name + local name (WebDAV, JCR)

This will require some work; expanded names may be a bit heavy-weight;
more feedback appreciated.

> 3. Content
>
> 3.1 A single binary stream per HTTP in WebDAV (ignoring content negotiation for
> now which is tricky in authoring); modeled as binary property in JCR (with only
> conventions on naming)

I think we have agreement that content *is* special and should not be
shoved into a property. It's not entirely clear whether we need to model
resources without content, and, if we do so, how we expose them over HTTP.

Best regards, Julian

Reply | Threaded
Open this post in threaded view
|

Re: Data Model

justinedelson
In reply to this post by David Nuescheler-3
On Wed, Sep 8, 2010 at 9:57 AM, David Nuescheler <[hidden email]> wrote:
>> If ordering of collection members is relevant, then mapping to a Javascript
>> array (instead of object) is probably required:
> sigh... i know... unfortunately...

Too bad this results in overly verbose data structures and annoyingly
complex iteration code.

Justin

>
> regards,
> david
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Data Model, as seen from WebDAV

Julian Reschke
In reply to this post by Julian Reschke
On 08.09.2010 17:17, Julian Reschke wrote:

> ...
>> 1.3 Identification: WebDAV: the name of a resource within a collection
>> is a
>> unique identifier. JCR: same (except that for same-name-siblings, the
>> array
>> index may change when siblings are removed/inserted).
>
> -> WebDAV model (where index notation would be used if we want same-name
> siblings)
> ...
>> 2.3 Naming: namespace name + local name (WebDAV, JCR)
>
> This will require some work; expanded names may be a bit heavy-weight;
> more feedback appreciated.
> ...

Related to naming of resources and properties we *also* have to think
about whether they should share the same space of names.

In WebDAV, properties are not exposed as HTTP resources (*), instead the
special HTTP method PROPFIND is used. Thus, collisions between
properties and children of a resource are no concern: a folder could
have both a child resource "foobar" and a property "foobar".

(*) Yes, that's one of the big issues with the WebDAV approach; but
maybe we can fix this by exposing WebDAV properties over HTTP GET in the
future.

In JCR, the same space of names is used for resources and properties,
thus once the name of a child resource and a property collide, it
depends on the type of API which one you'll get. This is something we
need to avoid.

[[ Note: XPath uses a path-like syntax to address both elements and
attributes, but easily works around this because @ is not valid in
element names; similarly, it takes advantage of other reserved
characters such as "(" and ")" to disambiguate between function and
element names ]]

In David's JSOP slides, we see examples similar to:

var collection1 = {

   "createdby" : "user1",

   "child1" : {

     "createdby" : "user2"

   },

   "child2" : {

     "createdby" : "user3"

   }
};

That has the same problem as JCR; member names and property names can
collide.

I think we'll need to disambiguate by introducing intermediary objects
such as "props" and "members", or even model the collection containment
using a reserved property name.

Best regards, Julian

Reply | Threaded
Open this post in threaded view
|

Re: Data Model, as seen from WebDAV

Mike Douglass
  In CalWS

http://www.calconnect.org/pubdocs/CD1011%20CalWS-Rest%20Restful%20Web%20Services%20Protocol%20for%20Calendaring%20V1.0.pdf

we exposed properties and relations using XRD

http://docs.oasis-open.org/xri/xrd/v1.0/xrd-1.0.html

which allows a GET with using ACCEPT to specify the xrd object

There's currently no way to specify which properties are desired.

> In WebDAV, properties are not exposed as HTTP resources (*), instead
> the special HTTP method PROPFIND is used. Thus, collisions between
> properties and children of a resource are no concern: a folder could
> have both a child resource "foobar" and a property "foobar".
>
> (*) Yes, that's one of the big issues with the WebDAV approach; but
> maybe we can fix this by exposing WebDAV properties over HTTP GET in
> the future.
>
> In JCR, the same space of names is used for resources and properties,
> thus once the name of a child resource and a property collide, it
> depends on the type of API which one you'll get. This is something we
> need to avoid.
>
> [[ Note: XPath uses a path-like syntax to address both elements and
> attributes, but easily works around this because @ is not valid in
> element names; similarly, it takes advantage of other reserved
> characters such as "(" and ")" to disambiguate between function and
> element names ]]
>
> In David's JSOP slides, we see examples similar to:
>
> var collection1 = {
>
>   "createdby" : "user1",
>
>   "child1" : {
>
>     "createdby" : "user2"
>
>   },
>
>   "child2" : {
>
>     "createdby" : "user3"
>
>   }
> };
>
> That has the same problem as JCR; member names and property names can
> collide.
>
> I think we'll need to disambiguate by introducing intermediary objects
> such as "props" and "members", or even model the collection
> containment using a reserved property name.
>
> Best regards, Julian
>
>

--

Mike Douglass                           [hidden email]
Senior Systems Programmer
Communication&  Collaboration Technologies      518 276 6780(voice) 2809
(fax)
Rensselaer Polytechnic Institute 110 8th Street, Troy, NY 12180


Reply | Threaded
Open this post in threaded view
|

Re: Data Model, as seen from WebDAV

Julian Reschke
On 20.09.2010 17:54, Mike Douglass wrote:

> In CalWS
>
> http://www.calconnect.org/pubdocs/CD1011%20CalWS-Rest%20Restful%20Web%20Services%20Protocol%20for%20Calendaring%20V1.0.pdf
>
>
> we exposed properties and relations using XRD
>
> http://docs.oasis-open.org/xri/xrd/v1.0/xrd-1.0.html
>
> which allows a GET with using ACCEPT to specify the xrd object
>
> There's currently no way to specify which properties are desired.
> ....

Content negotiation to get the metadata instead of the data?

Not sure this is a good idea...

Best regards, Julian

12