The Candidate Recommendation draft of SCXML has been published - call for implementation reports

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

The Candidate Recommendation draft of SCXML has been published - call for implementation reports

Jim Barnett
The new draft is available in the usual place:
http://www.w3.org/TR/scxml/   I would like to thank everyone whose
contributions and hard work have helped us get this far.

At this point there is only one (large) hurdle to cross before SCXML
becomes a full recommendation.  We need to show proof  of at least two
interoperable implementations of each feature.  The way we do this is by
providing a test suite and then showing that at least two
implementations pass each test.  The description of  the test suite and
links to the tests are here: http://www.w3.org/Voice/2013/scxml-irp/

The test suite is somewhat more complex than you might expect, because
we have decided to abstract away from the data model as well as the
means for recording success or failure.  Therefore the tests are not
written in straight scxml, but a mish-mash of scxml and elements in the
'txml' namespace.  You use an xslt style sheet to transform the txml
elements into scxml that matches your datamodel/implementation.  We
provide sample xslt files to produce the ECMAScript and XPath data
models, but you may edit them as you wish or provide your own (as long
as you don't change the underlying semantics of the test).

Please try to run the tests and submit an implementation report if you
can.  The more implementations that we get, the smoother our passage to
full recommendation status will be.  If you have any questions, don't
hesitate to contact me or to post to the list.  A couple of
implementations have already run many of the tests, but there may well
still be bugs in them or in the style sheets.

--
Jim Barnett
Genesys

P.S.  In terms of  W3C process, this is not a 'conformance test',
because the W3C does not do conformance testing.  However, it sure looks
like a conformance test to anyone who is not caught up in W3C process.  
In the case of the VoiceXML  specification, the VoiceXML Forum (which is
separate from the W3C) took the VXML implementation test suite and
produced an official conformance test from it.  There won't be an
official conformance test for SCXML unless an organization decides to do
the same.  Until then, if you pass the tests in the suite, your
implementation is conformant, but you can't say that you have passed a
"Conformance Test".

Reply | Threaded
Open this post in threaded view
|

Re: The Candidate Recommendation draft of SCXML has been published - call for implementation reports

David Junger
Some initial comments on the tests:

Tests 496 and 577 can be found if you just add the missing 'test' before the number.
Otherwise, 404 errors as Stefan said.

193: why does it now test with the basichttp IOproc instead of regular SCXML? the assertion on the IRP page doesn't say anything about that.
Same for 577. I suppose ONE of those is intended to test basichttp?

233: the XSLT produces cond="" which is always false (I'm using the same tool as Stefan).

286, 311, 331, 401, 402: I thought we agreed that, in the ECMAScript datamodel, assignment to an undeclared property of an existing object would NOT raise an error since ECMAScript itself is perfectly fine with that. The test should try assignment to e.g. "Var1.undefinedProperty" to guarantee an error.

286 and 311 seem to test the exact same thing.

557 and others: beside testing the actual assertion, these tests use <data src> which I have decided not to implement in JSSC. What I do in these cases is to remove the part that uses <data src> and use the result of the modified test, so you know if JSSC actually supports the tested feature (e.g. parsing XML data in 557), whereas using the original test would always fail and require a comment whose information is better expressed by the failure of the test that actually tests for <data src>.
Alternatively, I could rewrite the <data src> part to use <jssc:fetch>.
Anyway, I don't expect you to change the tests, just tell me what you'd prefer that I do about this.

561: the comment in the file and assertion on the IRP page say it tests JSON parsing, but the actual code still tests XML parsing, as in previous versions of this file.

                        David
Reply | Threaded
Open this post in threaded view
|

RE: The Candidate Recommendation draft of SCXML has been published - call for implementation reports

Jim Barnett-2

Yes, 193 and 577 are duplicates, both referring to section D.2.  There is no test for the corresponding passage in D.1, which is:  " If neither the 'target' nor the 'targetexpr' attribute is specified, the SCXML Processor MUST add the event to the external event queue of the sending session."  I will modify test193 to handle this case.

Test233 - fixed

Test286, etc.  The relevant language in the spec is in C.2.7: " When evaluating an <assign> element in the ECMAScript data model, the SCXML Processor MUST replace the existing value at 'location' with the value produced by evaluating 'expr'. If it is unable to do so (for example, if the <assign> element attempts to assign to a read-only attribute), it MUST place the error error.execution on the internal event queue."
In PySCXML, test286 passes as written.  Does this mean that Johan's implementation specifically tests for an undefined var and raises an error, or is there a difference in his ECMAScript implementation?  

I'll look at your other issues next.

- Jim

-----Original Message-----
From: David Junger [mailto:[hidden email]]
Sent: Saturday, March 15, 2014 8:14 AM
To: Voice Public List
Subject: Re: The Candidate Recommendation draft of SCXML has been published - call for implementation reports

Some initial comments on the tests:

Tests 496 and 577 can be found if you just add the missing 'test' before the number.
Otherwise, 404 errors as Stefan said.

193: why does it now test with the basichttp IOproc instead of regular SCXML? the assertion on the IRP page doesn't say anything about that.
Same for 577. I suppose ONE of those is intended to test basichttp?

233: the XSLT produces cond="" which is always false (I'm using the same tool as Stefan).

286, 311, 331, 401, 402: I thought we agreed that, in the ECMAScript datamodel, assignment to an undeclared property of an existing object would NOT raise an error since ECMAScript itself is perfectly fine with that. The test should try assignment to e.g. "Var1.undefinedProperty" to guarantee an error.

286 and 311 seem to test the exact same thing.

557 and others: beside testing the actual assertion, these tests use <data src> which I have decided not to implement in JSSC. What I do in these cases is to remove the part that uses <data src> and use the result of the modified test, so you know if JSSC actually supports the tested feature (e.g. parsing XML data in 557), whereas using the original test would always fail and require a comment whose information is better expressed by the failure of the test that actually tests for <data src>.
Alternatively, I could rewrite the <data src> part to use <jssc:fetch>.
Anyway, I don't expect you to change the tests, just tell me what you'd prefer that I do about this.

561: the comment in the file and assertion on the IRP page say it tests JSON parsing, but the actual code still tests XML parsing, as in previous versions of this file.

                        David



Reply | Threaded
Open this post in threaded view
|

Re: The Candidate Recommendation draft of SCXML has been published - call for implementation reports

David Junger
Le 17 mar 2014 à 16:25, Jim Barnett <[hidden email]> a écrit :

> Test286, etc.  The relevant language in the spec is in C.2.7: " When evaluating an <assign> element in the ECMAScript data model, the SCXML Processor MUST replace the existing value at 'location' with the value produced by evaluating 'expr'. If it is unable to do so (for example, if the <assign> element attempts to assign to a read-only attribute), it MUST place the error error.execution on the internal event queue."
> In PySCXML, test286 passes as written.  Does this mean that Johan's implementation specifically tests for an undefined var and raises an error, or is there a difference in his ECMAScript implementation?  

We arrived at the current wording some time ago after I raised the issue.
http://lists.w3.org/Archives/Public/www-voice/2013JanMar/0010.html

It is quite fundamental in ECMAScript, in fact the object-oriented part of it would be nearly impossible without assignments to undeclared properties.
I suppose Johan implemented the behavior described in earlier drafts and hasn't updated that part of PySCXML since the change.

                        David
Reply | Threaded
Open this post in threaded view
|

RE: The Candidate Recommendation draft of SCXML has been published - call for implementation reports

Jim Barnett-2
OK,  I've fixed tests 286, 311, 331, 401, 402 to refer to invalid locations in the ecma data model.  You'll have to reload the xslt file as well.

- Jim

-----Original Message-----
From: David Junger [mailto:[hidden email]]
Sent: Monday, March 17, 2014 1:12 PM
To: Voice Public List
Subject: Re: The Candidate Recommendation draft of SCXML has been published - call for implementation reports

Le 17 mar 2014 à 16:25, Jim Barnett <[hidden email]> a écrit :

> Test286, etc.  The relevant language in the spec is in C.2.7: " When evaluating an <assign> element in the ECMAScript data model, the SCXML Processor MUST replace the existing value at 'location' with the value produced by evaluating 'expr'. If it is unable to do so (for example, if the <assign> element attempts to assign to a read-only attribute), it MUST place the error error.execution on the internal event queue."
> In PySCXML, test286 passes as written.  Does this mean that Johan's implementation specifically tests for an undefined var and raises an error, or is there a difference in his ECMAScript implementation?  

We arrived at the current wording some time ago after I raised the issue.
http://lists.w3.org/Archives/Public/www-voice/2013JanMar/0010.html

It is quite fundamental in ECMAScript, in fact the object-oriented part of it would be nearly impossible without assignments to undeclared properties.
I suppose Johan implemented the behavior described in earlier drafts and hasn't updated that part of PySCXML since the change.

                        David



Reply | Threaded
Open this post in threaded view
|

RE: The Candidate Recommendation draft of SCXML has been published - call for implementation reports

Jim Barnett-2
In reply to this post by David Junger
As for tests 286 and 311, they do test they same thing, but they are based on two different assertions, one from section 5.4 and the other from section 5.9.  In general, we try to make every normative statement only once in the spec, but there are cases where a bit of repetition makes the spec easier to understand.  In such cases, we will end up with duplicate tests.

- Jim

-----Original Message-----
From: David Junger [mailto:[hidden email]]
Sent: Saturday, March 15, 2014 8:14 AM
To: Voice Public List
Subject: Re: The Candidate Recommendation draft of SCXML has been published - call for implementation reports

Some initial comments on the tests:

Tests 496 and 577 can be found if you just add the missing 'test' before the number.
Otherwise, 404 errors as Stefan said.

193: why does it now test with the basichttp IOproc instead of regular SCXML? the assertion on the IRP page doesn't say anything about that.
Same for 577. I suppose ONE of those is intended to test basichttp?

233: the XSLT produces cond="" which is always false (I'm using the same tool as Stefan).

286, 311, 331, 401, 402: I thought we agreed that, in the ECMAScript datamodel, assignment to an undeclared property of an existing object would NOT raise an error since ECMAScript itself is perfectly fine with that. The test should try assignment to e.g. "Var1.undefinedProperty" to guarantee an error.

286 and 311 seem to test the exact same thing.

557 and others: beside testing the actual assertion, these tests use <data src> which I have decided not to implement in JSSC. What I do in these cases is to remove the part that uses <data src> and use the result of the modified test, so you know if JSSC actually supports the tested feature (e.g. parsing XML data in 557), whereas using the original test would always fail and require a comment whose information is better expressed by the failure of the test that actually tests for <data src>.
Alternatively, I could rewrite the <data src> part to use <jssc:fetch>.
Anyway, I don't expect you to change the tests, just tell me what you'd prefer that I do about this.

561: the comment in the file and assertion on the IRP page say it tests JSON parsing, but the actual code still tests XML parsing, as in previous versions of this file.

                        David



Reply | Threaded
Open this post in threaded view
|

RE: The Candidate Recommendation draft of SCXML has been published - call for implementation reports

Jim Barnett-2
In reply to this post by David Junger
David,
  For test 557, I think it's fine for you to rewrite the test.  Using jssc:fetch might be the best way to do it, since that shows that you handle externally fetched data correctly (and the point of the test is to show that in-line and externally fetched data are handled the same way.)  However, when you get to the tests that check <data src=..> explicitly, you should record "not implemented" rather than reporting that you failed the test.

I've fixed test 561.  It is intended to test XML parsing.

- Jim

-----Original Message-----
From: David Junger [mailto:[hidden email]]
Sent: Saturday, March 15, 2014 8:14 AM
To: Voice Public List
Subject: Re: The Candidate Recommendation draft of SCXML has been published - call for implementation reports

Some initial comments on the tests:

Tests 496 and 577 can be found if you just add the missing 'test' before the number.
Otherwise, 404 errors as Stefan said.

193: why does it now test with the basichttp IOproc instead of regular SCXML? the assertion on the IRP page doesn't say anything about that.
Same for 577. I suppose ONE of those is intended to test basichttp?

233: the XSLT produces cond="" which is always false (I'm using the same tool as Stefan).

286, 311, 331, 401, 402: I thought we agreed that, in the ECMAScript datamodel, assignment to an undeclared property of an existing object would NOT raise an error since ECMAScript itself is perfectly fine with that. The test should try assignment to e.g. "Var1.undefinedProperty" to guarantee an error.

286 and 311 seem to test the exact same thing.

557 and others: beside testing the actual assertion, these tests use <data src> which I have decided not to implement in JSSC. What I do in these cases is to remove the part that uses <data src> and use the result of the modified test, so you know if JSSC actually supports the tested feature (e.g. parsing XML data in 557), whereas using the original test would always fail and require a comment whose information is better expressed by the failure of the test that actually tests for <data src>.
Alternatively, I could rewrite the <data src> part to use <jssc:fetch>.
Anyway, I don't expect you to change the tests, just tell me what you'd prefer that I do about this.

561: the comment in the file and assertion on the IRP page say it tests JSON parsing, but the actual code still tests XML parsing, as in previous versions of this file.

                        David



Reply | Threaded
Open this post in threaded view
|

Re: The Candidate Recommendation draft of SCXML has been published - call for implementation reports

Zjnue Brzavi
In reply to this post by Jim Barnett-2
 
Yes, 193 and 577 are duplicates, both referring to section D.2.  There is no test for the corresponding passage in D.1, which is:  " If neither the 'target' nor the 'targetexpr' attribute is specified, the SCXML Processor MUST add the event to the external event queue of the sending session."  I will modify test193 to handle this case.
 
Hi Jim,

I'm just catching up with this change and it appears the incorrect send type was specified for test 193.
Right now the type is specified as http://www.w3.org/TR/scxml/#BasicHTTPEventProcessor

However, the specification for <send> without target / targetexpr says the following:

Where type is BasicHTTPEventProcessor:
"If neither the 'target' nor the 'targetexpr' attribute is specified, the SCXML Processor MUST add the event error.communication to the internal event queue of the sending session."

Where type is SCXMLEventProcessor:
"If neither the 'target' nor the 'targetexpr' attribute is specified, the SCXML Processor MUST add the event to the external event queue of the sending session."

It appears test 193 is aiming to test the latter scenario, which means the send type should be SCXMLEventProcessor instead of BasicHTTPEventProcessor.

Best regards,

Zjnue

Reply | Threaded
Open this post in threaded view
|

RE: The Candidate Recommendation draft of SCXML has been published - call for implementation reports

Jim Barnett-2

Zjnue,

You are right.  I’ve fixed test 193 to use the SCXMLEventProcessor.

 

-          Jim

 

From: Zjnue Brzavi [mailto:[hidden email]]
Sent: Friday, March 21, 2014 6:01 PM
To: Jim Barnett
Cc: David Junger; Voice Public List
Subject: Re: The Candidate Recommendation draft of SCXML has been published - call for implementation reports

 

 

Yes, 193 and 577 are duplicates, both referring to section D.2.  There is no test for the corresponding passage in D.1, which is:  " If neither the 'target' nor the 'targetexpr' attribute is specified, the SCXML Processor MUST add the event to the external event queue of the sending session."  I will modify test193 to handle this case.

 

Hi Jim,

I'm just catching up with this change and it appears the incorrect send type was specified for test 193.

Right now the type is specified as http://www.w3.org/TR/scxml/#BasicHTTPEventProcessor

 

However, the specification for <send> without target / targetexpr says the following:

Where type is BasicHTTPEventProcessor:
"If neither the 'target' nor the 'targetexpr' attribute is specified, the SCXML Processor MUST add the event error.communication to the internal event queue of the sending session."

Where type is SCXMLEventProcessor:
"If neither the 'target' nor the 'targetexpr' attribute is specified, the SCXML Processor MUST add the event to the external event queue of the sending session."

It appears test 193 is aiming to test the latter scenario, which means the send type should be SCXMLEventProcessor instead of BasicHTTPEventProcessor.

Best regards,

Zjnue