Using wpt for CR exit

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Using wpt for CR exit

Geoffrey Sneddon-4
Hey,

Given as far as I know you've been at least somewhat involved with every
spec that's used wpt as part of its implementation report, let me throw
a few questions at you:

 * How's the lack of versioning been dealt with? (e.g., between HTML5
and tests for features added since HTML5 was forked)

 * How's the W3C/WHATWG divergence been dealt with? (For specs for which
that applies!)

Far as I can tell, neither of them really have been dealt with. If I'm
not misreading the few IRs I opened up, they've all assumed that all
tests within the respective directories in wpt apply to the spec leaving
CR. Furthermore, when it comes to links to the testsuite, they seem to
all just link to that directory in the master branch of wpt, which will
become increasingly less useful at testing a stable snapshot as time
progresses!

There's also the interesting case of a edit to a REC that ends up being
first published as a CR, where one would presumably want to add/modify
tests relative to the testsuite for the REC while excluding tests for
more recent revisions of the spec to exit CR then.

As an aside: there's been some discussion in the CSS WG about testsuites
recently (uh, my doing, "whoops") which you might have some interest in,
some of which deals with issues like testsuites for different versions
of a spec.

/Geoffrey

Reply | Threaded
Open this post in threaded view
|

Re: Using wpt for CR exit

Philippe Le Hegaret


On 03/29/2016 12:33 AM, Geoffrey Sneddon wrote:
> Hey,
>
> Given as far as I know you've been at least somewhat involved with every
> spec that's used wpt as part of its implementation report, let me throw
> a few questions at you:
>
>   * How's the lack of versioning been dealt with? (e.g., between HTML5
> and tests for features added since HTML5 was forked)

We used wptreport and filtering out test results we were not interested in:
  https://github.com/w3c/test-results/blob/gh-pages/html/filter.js

>   * How's the W3C/WHATWG divergence been dealt with? (For specs for which
> that applies!)

At the time of HTML 5.0, the diff between WHATWG and W3C was pretty much
the same as the diff between 5.0 and 5.1, so we just filtered out the
results that were only pertinent to 5.1. Other differences didn't have
impact on implementations.

I do realize that, while we were able to get away with this approach
back then, it wouldn't be so easy if an API was making a breaking change
such as changing the return type of a function.

> Far as I can tell, neither of them really have been dealt with. If I'm
> not misreading the few IRs I opened up, they've all assumed that all
> tests within the respective directories in wpt apply to the spec leaving
> CR. Furthermore, when it comes to links to the testsuite, they seem to
> all just link to that directory in the master branch of wpt, which will
> become increasingly less useful at testing a stable snapshot as time
> progresses!

We indeed didn't make a snapshot of the tests at the time. I guess if
one wanted to run the old html tests, one could checkout an old WPT
commit from that time.

> There's also the interesting case of a edit to a REC that ends up being
> first published as a CR, where one would presumably want to add/modify
> tests relative to the testsuite for the REC while excluding tests for
> more recent revisions of the spec to exit CR then.
>
> As an aside: there's been some discussion in the CSS WG about testsuites
> recently (uh, my doing, "whoops") which you might have some interest in,
> some of which deals with issues like testsuites for different versions
> of a spec.

I'll try to catch up on this front. I'm curious about the use cases for
the old test suites. While I can think of some, I'm worried about the
added costs for maintaining multiple versions of a test suite. We
already have such difficulties for maintaining multiple versions of a
spec...

Philippe

Reply | Threaded
Open this post in threaded view
|

Re: Using wpt for CR exit

Geoffrey Sneddon-4
On Tue, Mar 29, 2016 at 1:22 PM, Philippe Le Hegaret <[hidden email]> wrote:

> On 03/29/2016 12:33 AM, Geoffrey Sneddon wrote:
>> As an aside: there's been some discussion in the CSS WG about testsuites
>> recently (uh, my doing, "whoops") which you might have some interest in,
>> some of which deals with issues like testsuites for different versions
>> of a spec.
>
>
> I'll try to catch up on this front. I'm curious about the use cases for the
> old test suites. While I can think of some, I'm worried about the added
> costs for maintaining multiple versions of a test suite. We already have
> such difficulties for maintaining multiple versions of a spec...

I think the most obvious thing that comes to mind is the long period
it took to get CSS 2.1 out of CR, and the deferment of a number of
features to level 3. This meant punting a number of tests to their
respective level 3 modules (display: run-in comes to mind as something
that affected a lot of tests), while simultaneously maintaining the
2.1 testsuite for the sake of getting it out of CR. On the other hand,
things like that are almost certainly doable with simple filtering of
tests given an increasingly large blacklist can be used.

/Geoffrey