Re: Towards a better testsuite: Build System

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Build System

fantasai
On 03/24/2016 01:00 PM, Geoffrey Sneddon wrote:

>
> The other significant complication when it comes to csswg-test is the
> build system. Because of the necessity of building the testsuite first
> it makes it more complicated to fix tests when they're failing; you
> have to know how to find the source file and then be able to build the
> testsuite after fixing it. Historically the build system existed to
> deal with the variety in what UAs support, whether they supported HTML
> and/or XHTML; this is a far smaller deal nowadays—of the UAs running
> the CSS testsuite or likely to do so in the future, the only one I'm
> aware of that doesn't support both well is Servo (and that's likely to
> change so can possibly be ignored here).

I don't have much opinion on whether we keep or discard the
build system, but I don't think in any case that it should
be necessary in order for people to run or otherwise use the
tests. Tests are identified by filename, and run just fine
without the build system, so there's no need to build in the
general case.

For results reporting, we have Shepherd, which tracks which
tests belong to which suites, and doesn't much care where
they end up. The build system isn't necessary here, either.

However, individual vendors may need scripts to convert the
test-reference linkages into their preferred format E.g.
for Mozilla, we do need to generate reftest manifest files,
which are currently constructed by the build system. But
that can be done with a lighter-weight system that just
generates manifests in place per directory.

(As for adopting a "filename convention" for mapping the
tests and references... No. There are thousands of CSS tests
that use the same reference file. Whoever wants a "filename
convention" can make 1000 copies of each common reference if
they want, but I refuse to support such nonsense in the CSSWG
repository.)

~fantasai

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Build System

Ms2ger

On Apr 8, 2016 19:01, "fantasai" <[hidden email]> wrote:
> However, individual vendors may need scripts to convert the
> test-reference linkages into their preferred format E.g.
> for Mozilla, we do need to generate reftest manifest files,
> which are currently constructed by the build system. But
> that can be done with a lighter-weight system that just
> generates manifests in place per directory.

We don't, actually. We already run reftests from wpt using its manifest format; there's no reason to use reftest.list.

> (As for adopting a "filename convention" for mapping the
> tests and references... No. There are thousands of CSS tests
> that use the same reference file. Whoever wants a "filename
> convention" can make 1000 copies of each common reference if
> they want, but I refuse to support such nonsense in the CSSWG
> repository.)

Where did anybody suggest that? Wpt uses a filename convention to mark manual tests, but not for reftests.

Ms2ger

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Build System

fantasai
On 04/09/2016 07:42 AM, Ms2ger wrote:

> On Apr 8, 2016 19:01, "fantasai" <[hidden email] <mailto:[hidden email]>> wrote:
>> However, individual vendors may need scripts to convert the
>> test-reference linkages into their preferred format E.g.
>> for Mozilla, we do need to generate reftest manifest files,
>> which are currently constructed by the build system. But
>> that can be done with a lighter-weight system that just
>> generates manifests in place per directory.
>
> We don't, actually. We already run reftests from wpt using
> its manifest format; there's no reason to use reftest.list.

Right, but the CSSWG tests don't have any manifest; we use
<link> tags intead. So we'd need to generate some kind of
manifest, whether it's in WPT format or reftest.list format.

>> (As for adopting a "filename convention" for mapping the
>> tests and references... No. There are thousands of CSS tests
>> that use the same reference file. Whoever wants a "filename
>> convention" can make 1000 copies of each common reference if
>> they want, but I refuse to support such nonsense in the CSSWG
>> repository.)
>
> Where did anybody suggest that? Wpt uses a filename convention
> to mark manual tests, but not for reftests.

Well, apparently some people think this is a good idea:
   https://trac.webkit.org/wiki/Writing%20Reftests

p.s. why does your address book list public-css-testsuite as Mike Smith?

~fantasai

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Build System

Ms2ger

On Apr 9, 2016 4:32 PM, "fantasai" <[hidden email]> wrote:
>
> On 04/09/2016 07:42 AM, Ms2ger wrote:
>
>> On Apr 8, 2016 19:01, "fantasai" <[hidden email] <mailto:[hidden email]>> wrote:
>>>
>>> However, individual vendors may need scripts to convert the
>>> test-reference linkages into their preferred format E.g.
>>> for Mozilla, we do need to generate reftest manifest files,
>>> which are currently constructed by the build system. But
>>> that can be done with a lighter-weight system that just
>>> generates manifests in place per directory.
>>
>>
>> We don't, actually. We already run reftests from wpt using
>> its manifest format; there's no reason to use reftest.list.
>
>
> Right, but the CSSWG tests don't have any manifest; we use
> <link> tags intead. So we'd need to generate some kind of
> manifest, whether it's in WPT format or reftest.list format.

So does wpt; we went out of our way to use the exact same semantics for the link elements in wpt and csswg-test.

HTH
Ms2ger

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Build System

Geoffrey Sneddon-4
In reply to this post by fantasai
On Fri, Apr 8, 2016 at 6:00 PM, fantasai <[hidden email]> wrote:
> I don't have much opinion on whether we keep or discard the
> build system, but I don't think in any case that it should
> be necessary in order for people to run or otherwise use the
> tests. Tests are identified by filename, and run just fine
> without the build system, so there's no need to build in the
> general case.

For those who don't read minutes: on the telecon I took ACTION-766 to
essentially discard the build system.

https://bitbucket.org/gsnedders/w3ctestlib/commits/d53d2407c01fc9ce62b68c7256b709c77d4d2a04
achieves this. It'd be great to get this reviewed and landed, Peter,
along with everything else on my fork of w3ctestlib.

/Geoffrey

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Build System

Peter Linss

> On Apr 14, 2016, at 9:47 PM, Geoffrey Sneddon <[hidden email]> wrote:
>
> On Fri, Apr 8, 2016 at 6:00 PM, fantasai <[hidden email]> wrote:
>> I don't have much opinion on whether we keep or discard the
>> build system, but I don't think in any case that it should
>> be necessary in order for people to run or otherwise use the
>> tests. Tests are identified by filename, and run just fine
>> without the build system, so there's no need to build in the
>> general case.
>
> For those who don't read minutes: on the telecon I took ACTION-766 to
> essentially discard the build system.
The action wasn’t to “discard” the build system, it was to make building un-necessary to run the tests. As I said on the call, we have infrastructure (the harness, the spec annotation system, et al) that currently relies on the build output. I thought it was fairly clear that we’d still be running the existing build system on the server until the infrastructure gets updated to not rely on it.

>
> https://bitbucket.org/gsnedders/w3ctestlib/commits/d53d2407c01fc9ce62b68c7256b709c77d4d2a04
> achieves this. It'd be great to get this reviewed and landed, Peter,
> along with everything else on my fork of w3ctestlib.

We can’t land this change, as it would totally break the test harness and annotation system.

A workable approach is to leave the existing build system in place (including the index pages), but have a switch that makes the build script simply build the manifest file(s) instead. I’m fine if the new behavior is the default as I can add the switch to the server’s build process quickly enough.

FWIW, people have used the implementation report templates so we should keep those as well.

Peter

signature.asc (506 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Build System

Geoffrey Sneddon-4
On Mon, Apr 18, 2016 at 12:28 AM, Peter Linss <[hidden email]> wrote:

>
>> On Apr 14, 2016, at 9:47 PM, Geoffrey Sneddon <[hidden email]> wrote:
>>
>> On Fri, Apr 8, 2016 at 6:00 PM, fantasai <[hidden email]> wrote:
>>> I don't have much opinion on whether we keep or discard the
>>> build system, but I don't think in any case that it should
>>> be necessary in order for people to run or otherwise use the
>>> tests. Tests are identified by filename, and run just fine
>>> without the build system, so there's no need to build in the
>>> general case.
>>
>> For those who don't read minutes: on the telecon I took ACTION-766 to
>> essentially discard the build system.
>
> The action wasn’t to “discard” the build system, it was to make building un-necessary to run the tests. As I said on the call, we have infrastructure (the harness, the spec annotation system, et al) that currently relies on the build output. I thought it was fairly clear that we’d still be running the existing build system on the server until the infrastructure gets updated to not rely on it.

OK, I obviously hadn't understood that our infrastructure was
necessarily a blocker. :)

What I think was clear from the call is that outside of the WG's
infrastructure, nothing relied upon the build system. IMO, it adds
sufficient complexity to authoring tests (how many long-term
contributors have fallen foul of unique-filename restriction, for
example) that ideally we should just discard it if it affects nobody
outside of the WG, and we should just fix our infrastructure.

How much of the infrastructure actually relies on it? Taking a look
around the test harness source code I found very little, but I could
just be missing things (I after all don't really understand much of
how it works, especially given its dearth of documentation). It seems
like it shouldn't be too much work to get it working with the unbuilt
testsuite if we introduce a new "unbuilt" format.

That said, quite how it builds the testsuite and populates the
database is opaque to me: there certainly doesn't appear to be
anything that runs build.py in any of the server apps repositories, so
I could be missing some fairly complex part of the system that would
need updating!

Lastly, just to make sure I'm not missing anything: the spec
annotation system is that which adds the "x tests" annotation to
specs, loading a script and data from the test harness?

/Geoffrey