URI Template -04 Implementation Notes

[Joe Gregorio-2]

I've committed an incomplete implementation of the -04 spec
in Python to svn along with test cases taken from the spec.

   http://code.google.com/p/uri-templates/source/detail?r=57

The tests themselves are taken directly from the spec and
are stored in a JSON file, to make it easier to reuse them.
(Well, almost taken verbatim from the spec, below I list
the cases that were either inconsistent or looked wrong
and how I changed them.) The code does not implement
'partial' modifiers.

Here are my notes:

1. The spec is fiddly, by which I mean that while the rules
   for handling semicolon, query parameter, path and label
   expansions all make sense on their own, they are all
   slightly different from each other. It makes for a lot
   more code than I expected, and is fairly tedious to implement.
   I'm not sure that the amount of complexity is warranted.

   To simplify I would like to see label and semicolon path
   style expansions dropped. I also believe the '+' modifier could
   be dropped without loss of generality.

2. From the spec, this rule isn't consistent:
      ["{?x,y,empty}", "?x=1024&y=768&empty="],

3. One of these is inconsistent
     ["X{.empty}", "X."],
     ["x{?empty_list}", "x"],

4. Also inconsistent
     ["{;list}", ";val1,val2,val3"],
     ["x{;name=none}", "x;name=Fred,Wilma,Pebbles"],

5. I tried working through the empty and default cases
    (section 2.5) but came across so many cases that
    seemed to disagree with previous examples or the wording
    of the spec that I began to suspect the section was incomplete.
    Here is the list of test cases I had collected and believe are wrong,
    before I stopped:

     ["x{empty_list=_}y", "xy"],
     ["x{empty_list+=_}y", "xempty_list._y"],
     ["x{empty_keys=_}y", "xy"],
     ["x{empty_keys+=_}y", "xempty_keys._y"],
     ["x{?empty}", "x?empty="],
     ["x{?empty=none}", "x?empty="],
     ["x{?empty_list*=none}", "x?none"],
     ["x{?empty_list+=none}", "x?empty_list.none"],
     ["x{?empty_keys*=none}", "x?none"],
     ["x{?empty_keys+=none}", "x?empty_keys.none"],

  I believe many of these cases with defaults are ones that Mike Burrows
  raised earlier.

6. I've come across yet another case where going in reverse, from a template
   to matching, would be useful. How about carving out () at the end of
   an expression as a place where a regex could be stored for applications
   that want to use a template for matching against incoming requests, but
   just state that the allowed regex's and matching algorithms are outside
   the scope of the current spec?

  Thanks,
  -joe

[Kev Burns]

In implementing a -04 parser in PHP, I also found the expansion operators to treat modifiers dissimilarly, resulting in a large number of special cases.
http://lab.kevburnsjr.com/php-uri-template-parser

I did, however, get the library to pass for all examples I could find in the -04 spec (except those relating to partial modifiers which I have yet to implement).
http://php-uri-template-parser.hackyhack.net/tests.php


KB - a
- --- -
> 1. [...]
>
>   To simplify I would like to see label and semicolon path
>   style expansions dropped. I also believe the '+' modifier could
>   be dropped without loss of generality.

In the step from -03 to -04 the syntax has become more succinct, but has also become more magical.  Simpler is better IMHO.


KB - b
- --- -
> 6. I've come across yet another case where going in reverse, from a
>  template to matching, would be useful [...]

Reversibility was something that originally attracted me to URI-Templates.  If a resource could use the same pattern to match a URI as it uses to generate them, the world could be a much simpler place. 

However, it could be argued that most templates are not designed to be reversible (you wouldn't try to reverse an HTML template expansion).

The best solution I could come up with is a second format employing an extended expression syntax (adding support for RegEx) used for matching that could be easily reduced to a valid uri-template for expansion.  A reversible-uri-template if you will. 

Keeping this as a separate format would ensure that uri-template parsers remain simple to implement (important for adoption IMHO).


KB - c
- --- -
Lastly, I wondered about using multiple operators for a single expression, particularly with respect to the + operator.

Should expressions with multiple operators such as {+;list1,list2} be allowed in order to add support for Reserved Expansion of expressions with delimiter operators?


- KevBurnsJr
c: +001 [phone]

On Tue, Jun 1, 2010 at 9:00 PM, Joe Gregorio <[hidden email]> wrote:

I've committed an incomplete implementation of the -04 spec
in Python to svn along with test cases taken from the spec.

  http://code.google.com/p/uri-templates/source/detail?r=57

The tests themselves are taken directly from the spec and
are stored in a JSON file, to make it easier to reuse them.
(Well, almost taken verbatim from the spec, below I list
the cases that were either inconsistent or looked wrong
and how I changed them.) The code does not implement
'partial' modifiers.

Here are my notes:

1. The spec is fiddly, by which I mean that while the rules
  for handling semicolon, query parameter, path and label
  expansions all make sense on their own, they are all
  slightly different from each other. It makes for a lot
  more code than I expected, and is fairly tedious to implement.
  I'm not sure that the amount of complexity is warranted.

  To simplify I would like to see label and semicolon path
  style expansions dropped. I also believe the '+' modifier could
  be dropped without loss of generality.

2. From the spec, this rule isn't consistent:
     ["{?x,y,empty}", "?x=1024&y=768&empty="],

3. One of these is inconsistent
    ["X{.empty}", "X."],
    ["x{?empty_list}", "x"],

4. Also inconsistent
    ["{;list}", ";val1,val2,val3"],
    ["x{;name=none}", "x;name=Fred,Wilma,Pebbles"],

5. I tried working through the empty and default cases
   (section 2.5) but came across so many cases that
   seemed to disagree with previous examples or the wording
   of the spec that I began to suspect the section was incomplete.
   Here is the list of test cases I had collected and believe are wrong,
   before I stopped:

    ["x{empty_list=_}y", "xy"],
    ["x{empty_list+=_}y", "xempty_list._y"],
    ["x{empty_keys=_}y", "xy"],
    ["x{empty_keys+=_}y", "xempty_keys._y"],
    ["x{?empty}", "x?empty="],
    ["x{?empty=none}", "x?empty="],
    ["x{?empty_list*=none}", "x?none"],
    ["x{?empty_list+=none}", "x?empty_list.none"],
    ["x{?empty_keys*=none}", "x?none"],
    ["x{?empty_keys+=none}", "x?empty_keys.none"],

 I believe many of these cases with defaults are ones that Mike Burrows
 raised earlier.

6. I've come across yet another case where going in reverse, from a template
  to matching, would be useful. How about carving out () at the end of
  an expression as a place where a regex could be stored for applications
  that want to use a template for matching against incoming requests, but
  just state that the allowed regex's and matching algorithms are outside
  the scope of the current spec?

 Thanks,
 -joe

On 2 June 2010 05:00, Joe Gregorio <[hidden email]> wrote:

I've committed an incomplete implementation of the -04 spec
in Python to svn along with test cases taken from the spec.

  http://code.google.com/p/uri-templates/source/detail?r=57

The tests themselves are taken directly from the spec and
are stored in a JSON file, to make it easier to reuse them.
(Well, almost taken verbatim from the spec, below I list
the cases that were either inconsistent or looked wrong
and how I changed them.) The code does not implement
'partial' modifiers.

Cool (the JSON part especially).
 

Here are my notes:

1. The spec is fiddly, by which I mean that while the rules
  for handling semicolon, query parameter, path and label
  expansions all make sense on their own, they are all
  slightly different from each other. It makes for a lot
  more code than I expected, and is fairly tedious to implement.
  I'm not sure that the amount of complexity is warranted.

 

  To simplify I would like to see label and semicolon path
  style expansions dropped. I also believe the '+' modifier could
  be dropped without loss of generality.

Please don't drop label expansion.  I added this to Routes (as used by Pylons) and it enables applications to simplify their routing quite significantly.  It's the optionality that makes them not redundant (unless you want to revive the prefix operator!).  I'm not bothered by the other features, but are there more consistent ways of specifying them?

But yes, it was fiddly and annoying.

 

2. From the spec, this rule isn't consistent:
     ["{?x,y,empty}", "?x=1024&y=768&empty="],

Yes, it seems we have both concluded that the trailing '=' above is needed (missing in the spec).

 

3. One of these is inconsistent
    ["X{.empty}", "X."],
    ["x{?empty_list}", "x"],

My implementation manages both of these.  Perhaps (it was a while ago) it was at the cost of a special case in the code though.  What would you change them to?

 

4. Also inconsistent
    ["{;list}", ";val1,val2,val3"],
    ["x{;name=none}", "x;name=Fred,Wilma,Pebbles"],

Yes, I disabled the latter test.

 

5. I tried working through the empty and default cases
   (section 2.5) but came across so many cases that
   seemed to disagree with previous examples or the wording
   of the spec that I began to suspect the section was incomplete.
   Here is the list of test cases I had collected and believe are wrong,
   before I stopped:

    ["x{empty_list=_}y", "xy"],
    ["x{empty_list+=_}y", "xempty_list._y"],
    ["x{empty_keys=_}y", "xy"],
    ["x{empty_keys+=_}y", "xempty_keys._y"],
    ["x{?empty}", "x?empty="],
    ["x{?empty=none}", "x?empty="],
    ["x{?empty_list*=none}", "x?none"],
    ["x{?empty_list+=none}", "x?empty_list.none"],
    ["x{?empty_keys*=none}", "x?none"],
    ["x{?empty_keys+=none}", "x?empty_keys.none"],

 I believe many of these cases with defaults are ones that Mike Burrows
 raised earlier.

I pass several of those; I got away with disabling out just these tests:


    ('x{empty_list=_}y',       'xy'),
    ('x{empty_keys=_}y',       'xy'),
    ('x{;name=none}',          'x;name=Fred,Wilma,Pebbles'),
    ('x{;favs=none}',          'x;favs=color,red,volume,high'),
    ('x{;empty_list=none}',    'x;empty_list=none'),
    ('x{;empty_keys=none}',    'x;empty_keys=none'),

 

6. I've come across yet another case where going in reverse, from a template
  to matching, would be useful. How about carving out () at the end of
  an expression as a place where a regex could be stored for applications
  that want to use a template for matching against incoming requests, but
  just state that the allowed regex's and matching algorithms are outside
  the scope of the current spec?

That could be done without changing the spec at this stage.  Routes (for example) has a syntax that allows a regex to be embedded - e.g. "{foo:[a-z]+}" - but it's usually much better to provide these as separate parameters to the route definition.  Implementers are free to do something similar now with the URI Template spec as it stands, and meanwhile the spec can progress towards approval.

It should be sufficient for servers to be able to generate URI Templates from their native formats (this is what described_routes does with templates from Routes or Rails); in the worst case they can be maintained separately.  So far I've manage to avoid doing anything that will encourage clients to parse URIs.

In summary, there are only a few broken tests, the complexity is slightly annoying but still manageable (assuming the broken tests can be fixed to something sensible), and I for one would still press on towards approval.  Meanwhile, feel free to get what you can from

    http://bitbucket.org/asplake/described_routes/src/tip/described_routes/uri_template.py
    http://bitbucket.org/asplake/described_routes/src/tip/tests/test_uri_template.py

When the spec is stable it would be nice to release either package (I really don't mind whose) to pypi.

 Thanks,
 -joe

Regards,
Mike
[hidden email]
http://positiveincline.com
http://twitter.com/asplake

Further to point 5, these are the actual expansions given by my implementation:

x{empty_list=_}y -> x_y (was: xy)
x{empty_keys=_}y -> x_y (was: xy)
x{;name=none} -> x;Fred,Wilma,Pebbles (was: x;name=Fred,Wilma,Pebbles)
x{;favs=none} -> x;color,red,volume,high (was: x;favs=color,red,volume,high)
x{;empty_list=none} -> x;none (was: x;empty_list=none)
x{;empty_keys=none} -> x;none (was: x;empty_keys=none)

Mike
[hidden email]
http://positiveincline.com
http://twitter.com/asplake

On 2 June 2010 11:18, Mike Burrows <[hidden email]> wrote:

On 2 June 2010 05:00, Joe Gregorio <[hidden email]> wrote:

I've committed an incomplete implementation of the -04 spec
in Python to svn along with test cases taken from the spec.

  http://code.google.com/p/uri-templates/source/detail?r=57

The tests themselves are taken directly from the spec and
are stored in a JSON file, to make it easier to reuse them.
(Well, almost taken verbatim from the spec, below I list
the cases that were either inconsistent or looked wrong
and how I changed them.) The code does not implement
'partial' modifiers.

Cool (the JSON part especially).
 

Here are my notes:

1. The spec is fiddly, by which I mean that while the rules
  for handling semicolon, query parameter, path and label
  expansions all make sense on their own, they are all
  slightly different from each other. It makes for a lot
  more code than I expected, and is fairly tedious to implement.
  I'm not sure that the amount of complexity is warranted.

 

  To simplify I would like to see label and semicolon path
  style expansions dropped. I also believe the '+' modifier could
  be dropped without loss of generality.

Please don't drop label expansion.  I added this to Routes (as used by Pylons) and it enables applications to simplify their routing quite significantly.  It's the optionality that makes them not redundant (unless you want to revive the prefix operator!).  I'm not bothered by the other features, but are there more consistent ways of specifying them?

But yes, it was fiddly and annoying.

 

2. From the spec, this rule isn't consistent:
     ["{?x,y,empty}", "?x=1024&y=768&empty="],

Yes, it seems we have both concluded that the trailing '=' above is needed (missing in the spec).

 

3. One of these is inconsistent
    ["X{.empty}", "X."],
    ["x{?empty_list}", "x"],

My implementation manages both of these.  Perhaps (it was a while ago) it was at the cost of a special case in the code though.  What would you change them to?

 

4. Also inconsistent
    ["{;list}", ";val1,val2,val3"],
    ["x{;name=none}", "x;name=Fred,Wilma,Pebbles"],

Yes, I disabled the latter test.

 

5. I tried working through the empty and default cases
   (section 2.5) but came across so many cases that
   seemed to disagree with previous examples or the wording
   of the spec that I began to suspect the section was incomplete.
   Here is the list of test cases I had collected and believe are wrong,
   before I stopped:

    ["x{empty_list=_}y", "xy"],
    ["x{empty_list+=_}y", "xempty_list._y"],
    ["x{empty_keys=_}y", "xy"],
    ["x{empty_keys+=_}y", "xempty_keys._y"],
    ["x{?empty}", "x?empty="],
    ["x{?empty=none}", "x?empty="],
    ["x{?empty_list*=none}", "x?none"],
    ["x{?empty_list+=none}", "x?empty_list.none"],
    ["x{?empty_keys*=none}", "x?none"],
    ["x{?empty_keys+=none}", "x?empty_keys.none"],

 I believe many of these cases with defaults are ones that Mike Burrows
 raised earlier.

I pass several of those; I got away with disabling out just these tests:


    ('x{empty_list=_}y',       'xy'),
    ('x{empty_keys=_}y',       'xy'),
    ('x{;name=none}',          'x;name=Fred,Wilma,Pebbles'),
    ('x{;favs=none}',          'x;favs=color,red,volume,high'),
    ('x{;empty_list=none}',    'x;empty_list=none'),
    ('x{;empty_keys=none}',    'x;empty_keys=none'),

 

6. I've come across yet another case where going in reverse, from a template
  to matching, would be useful. How about carving out () at the end of
  an expression as a place where a regex could be stored for applications
  that want to use a template for matching against incoming requests, but
  just state that the allowed regex's and matching algorithms are outside
  the scope of the current spec?

That could be done without changing the spec at this stage.  Routes (for example) has a syntax that allows a regex to be embedded - e.g. "{foo:[a-z]+}" - but it's usually much better to provide these as separate parameters to the route definition.  Implementers are free to do something similar now with the URI Template spec as it stands, and meanwhile the spec can progress towards approval.

It should be sufficient for servers to be able to generate URI Templates from their native formats (this is what described_routes does with templates from Routes or Rails); in the worst case they can be maintained separately.  So far I've manage to avoid doing anything that will encourage clients to parse URIs.

In summary, there are only a few broken tests, the complexity is slightly annoying but still manageable (assuming the broken tests can be fixed to something sensible), and I for one would still press on towards approval.  Meanwhile, feel free to get what you can from

    http://bitbucket.org/asplake/described_routes/src/tip/described_routes/uri_template.py
    http://bitbucket.org/asplake/described_routes/src/tip/tests/test_uri_template.py

When the spec is stable it would be nice to release either package (I really don't mind whose) to pypi.

 Thanks,
 -joe

Regards,

Mike
[hidden email]

http://positiveincline.com
http://twitter.com/asplake

Hi Joe,

Re point 6, just to mention that there are examples of what I describe in the JSON description of the Buzz API.  I am led to believe that you are familiar with this.

That aside, where next with the spec, implementation issues etc?  If there's anything I can do...

Regards,
Mike

[hidden email]
http://positiveincline.com
http://twitter.com/asplake


On 2 June 2010 11:18, Mike Burrows <[hidden email]> wrote:

<snip> 

 

On 2 June 2010 05:00, Joe Gregorio <[hidden email]> wrote: 

6. I've come across yet another case where going in reverse, from a template
  to matching, would be useful. How about carving out () at the end of
  an expression as a place where a regex could be stored for applications
  that want to use a template for matching against incoming requests, but
  just state that the allowed regex's and matching algorithms are outside
  the scope of the current spec?

That could be done without changing the spec at this stage.  Routes (for example) has a syntax that allows a regex to be embedded - e.g. "{foo:[a-z]+}" - but it's usually much better to provide these as separate parameters to the route definition.  Implementers are free to do something similar now with the URI Template spec as it stands, and meanwhile the spec can progress towards approval.

It should be sufficient for servers to be able to generate URI Templates from their native formats (this is what described_routes does with templates from Routes or Rails); in the worst case they can be maintained separately.  So far I've manage to avoid doing anything that will encourage clients to parse URIs.

<snip>

[Roy T. Fielding]

I am trying to reconcile URI Templates with the apparent lack
of consensus on the more complex features.  I guess that confusion
shouldn't be surprising, since I forgot that I hadn't included
definitions for each of the operators in draft 04 (sections 3.7-10),
and then all my spec plans were discarded when my son was born the
following week.

One of the big problems seems to be wherever template variables
make reference to implementation language types, like lists or
structures, where everyone seems to have a different opinion on
how to handle empty (or zero member) values.  Likewise, I thought
that the explode syntax would be valuable for expressing many-variable
templates in a very short string, but it seems to add too much
complexity to the implementations.

On Jun 1, 2010, at 9:00 PM, Joe Gregorio wrote:

> I've committed an incomplete implementation of the -04 spec
> in Python to svn along with test cases taken from the spec.
>
>   http://code.google.com/p/uri-templates/source/detail?r=57
>
> The tests themselves are taken directly from the spec and
> are stored in a JSON file, to make it easier to reuse them.
> (Well, almost taken verbatim from the spec, below I list
> the cases that were either inconsistent or looked wrong
> and how I changed them.) The code does not implement
> 'partial' modifiers.

Can you say why it does not implement partials?  That should
be one of the easiest things to implement, and yet two people
didn't bother.

> Here are my notes:
>
> 1. The spec is fiddly, by which I mean that while the rules
>   for handling semicolon, query parameter, path and label
>   expansions all make sense on their own, they are all
>   slightly different from each other. It makes for a lot
>   more code than I expected, and is fairly tedious to implement.
>   I'm not sure that the amount of complexity is warranted.

The goal is to make them consistent with the way that the
same syntax is handled on websites.  Semicolon, query, labels,
and paths have distinct parsing behavior on websites regardless
of templates.

Looking at your implementation, I think the tedious bit is due
to the special handling of lists and modifiers, not the different
behavior of operators.

>   To simplify I would like to see label and semicolon path
>   style expansions dropped. I also believe the '+' modifier could
>   be dropped without loss of generality.

I don't think that makes any sense.  Label and params are the
two most common variations on path selectors (e.g., Apache Sling,
Rails, etc.).  Having them as operators allows the ";" or "." prefix
to be elided when optional, and multiple values to be properly separated.
Trying to define URI templates without them will just eliminate
templates as a viable description for those resources.

> 2. From the spec, this rule isn't consistent:
>      ["{?x,y,empty}", "?x=1024&y=768&empty="],

It is consistent with HTML forms.  When a form variable is empty, it
is submitted as name=

> 3. One of these is inconsistent
>     ["X{.empty}", "X."],
>     ["x{?empty_list}", "x"],

I made a distinction between empty string and empty list.  I should
just remove lists and exclude empty labels.

> 4. Also inconsistent
>     ["{;list}", ";val1,val2,val3"],
>     ["x{;name=none}", "x;name=Fred,Wilma,Pebbles"],

Yes, the second was intended.

> 5. I tried working through the empty and default cases
>    (section 2.5) but came across so many cases that
>    seemed to disagree with previous examples or the wording
>    of the spec that I began to suspect the section was incomplete.
>    Here is the list of test cases I had collected and believe are wrong,
>    before I stopped:
>
>     ["x{empty_list=_}y", "xy"],
>     ["x{empty_list+=_}y", "xempty_list._y"],
>     ["x{empty_keys=_}y", "xy"],
>     ["x{empty_keys+=_}y", "xempty_keys._y"],
>     ["x{?empty}", "x?empty="],
>     ["x{?empty=none}", "x?empty="],
>     ["x{?empty_list*=none}", "x?none"],
>     ["x{?empty_list+=none}", "x?empty_list.none"],
>     ["x{?empty_keys*=none}", "x?none"],
>     ["x{?empty_keys+=none}", "x?empty_keys.none"],
>
>  I believe many of these cases with defaults are ones that Mike Burrows
>  raised earlier.

Yes, they are all due to the distinction between empty strings and
empty lists, and when lists are expanded versus just treated as another
string.  The missing prose doesn't help, but the easier solution
is to remove those as options.

> 6. I've come across yet another case where going in reverse, from a template
>   to matching, would be useful. How about carving out () at the end of
>   an expression as a place where a regex could be stored for applications
>   that want to use a template for matching against incoming requests, but
>   just state that the allowed regex's and matching algorithms are outside
>   the scope of the current spec?

I don't think so.  A regex can be derived from the template if the expressions
are delimited by reserved characters and reserved substitution is not used.
If those two conditions are not true, then a matching regex would have to be
ambiguous, which won't satisfy anyone.  I think we should just say what
people should limit their template to if they want it to be matching.

I am also leaning toward making the reserved expansion "+" operator
a prefix on any variable, since that would be backward compatible with
simple expressions and yet give people the rope they want to hang other
values on without complicating the implementations further.  We can then
move the encode-reserved question inside the value-substitution routine
rather than being operator-specific.

....Roy