Suggestion to re-consider exit criterion for CT guidelines

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Suggestion to re-consider exit criterion for CT guidelines

Francois Daoust
Dear BPWG participants,

Some news about the transition request for the Guidelines for Web
Content Transformation Proxies to Candidate Recommendation.

On internal review of the transition request there was push back on the
absence of a test case repository. Accordingly, I want to re-raise the
proposal to bind exit criteria to passing tests of a yet-to-exist test
suite.

More specifically, below is a list of suggestions that I think the group
should consider, and adopt if possible. They would all affect the Status
Of This Document (SOTD) section of the guidelines. Latest version of
this SOTD, that I had prepared for the transition in accordance with the
group's resolutions, is available at:
 
http://www.w3.org/2005/MWI/BPWG/Group/TaskForces/CT/editors-drafts/Guidelines/CR-ct-guidelines-20100427/#status


1. Link to a test case repository
-----
I had actually prepared a test case repository on W3C CVS servers along
with a document that references the existing test cases:
  http://dev.w3.org/2010/ct-guidelines/test-suite/

First idea would be to say that this test case repository exists and
link to that document from the SOTD section.



2. Nature of the test suite
-----
Not all the guidelines can be tested, and creating a complete test suite
for those who can be is not easy either. We cannot just come up with a
test suite that ensures that a proxy follows the HTTP RFC!

Thus, second idea would be to precise that the test suite is not
intended to cover each and every aspect of each and every guideline, but
is more to be viewed as a collection of challenge test cases on
assertions for which such challenges may be created.

For instance, it is easy to write challenges to test assertions such as
"[Proxies] MUST NOT delete header fields" (ta-13875). On the contrary,
it is basically impossible to write test cases for "Proxies MUST assess
whether a user's expressed preference for a restructured representation
is still valid ..." (ta-38061).

Clarifying the fact that the test suite cannot be complete and is mostly
to contain challenge test cases would make it possible:
- to ask for contributions from the community
- to note guidelines that cannot be tested and whose implementation can
but rely on good faith assertions from implementers
- to actually finish the test suite!

Note that working on the test suite does not mean that the group needs
to do the testing itself.



3. Commitment to work on the test suite
-----
The repository should at least contain one test case before the
transition occurs.
The group should also commit to best effort to add tests to the
repository during the Candidate Recommendation phase, but I think we
already agreed that this latter point was a priority once the transition
occur.



4. Bind exit criteria to test suite
-----
Once The test case repository has been identified and clarified in terms
of content, final idea would be to amend the exit criterion and add the
fact that a claim of conformance involves passing the challenge tests of
the test repository, completed by good faith assertions that each of the
mandatory requirements has been implemented. It is a combination
approach between the group's current position and a position based on a
complete test suite which we know cannot be achieved.


Dan, Jo, could you add this to next call's agenda?
(also, next call is on May 11th, should we rather hold a call next week
on May 4th?)


Thanks,
Francois.