Getting browser vendors running and submitting tests

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Getting browser vendors running and submitting tests

Geoffrey Sneddon-4
[CC'd to public-test-infra; this is really mostly about CSS, can we avoid cross-posting replies and keeping them on public-css-testsuite?]

Myself and fantasai chaired a session at TPAC earlier today on this subject. The minutes are at <http://www.w3.org/2015/10/28-testing-minutes.html> (thanks fantasai!).

Notably:

* The build system makes it difficult for vendors to upstream their fixes and their own tests (as vendors can't just fix the files they run; they have to find some other file, fix that, and then work out how to build the test suite). As a result, they just fix the built files and don't upstream the fixes and often avoid updating the copy of the test suite because it would trample their fixes. We should get rid of the build system (at least insofar as it modifies/moves files), as I don't think anyone likely to run the tests doesn't support HTML at this point (yay the HTML5 parser!).

* We should get rid of metadata, at least in the common case, required to submit tests to the the test suite, as this is an impediment to upstreaming tests. (See also my recent thread on public-css-testsuite, "Simplifying metadata".)

These would make it far easier for browsers to run the up-to-date CSS test suite in their CI systems, to push fixes to the test suite, and to release tests their have into the test suite.

/gsnedders
Reply | Threaded
Open this post in threaded view
|

Re: Getting browser vendors running and submitting tests

Gérard Talbot-3
Le 2015-10-28 04:21, Geoffrey Sneddon a écrit :

> [CC'd to public-test-infra; this is really mostly about CSS, can we
> avoid
> cross-posting replies and keeping them on public-css-testsuite?]
>
> Myself and fantasai chaired a session at TPAC earlier today on this
> subject. The minutes are at <
> http://www.w3.org/2015/10/28-testing-minutes.html> (thanks fantasai!).
>
> Notably:
>
> * The build system makes it difficult for vendors to upstream their
> fixes
> and their own tests (as vendors can't just fix the files they run; they
> have to find some other file, fix that, and then work out how to build
> the
> test suite). As a result, they just fix the built files and don't
> upstream
> the fixes and often avoid updating the copy of the test suite because
> it
> would trample their fixes.

Why they don't fix their submitted tests (their tests in
http://test.csswg.org/source/ ) and then re-import the modified test
suite in their house system ?

Gérard

> We should get rid of the build system (at least
> insofar as it modifies/moves files), as I don't think anyone likely to
> run
> the tests doesn't support HTML at this point (yay the HTML5 parser!).
>
> * We should get rid of metadata, at least in the common case, required
> to
> submit tests to the the test suite, as this is an impediment to
> upstreaming
> tests. (See also my recent thread on public-css-testsuite, "Simplifying
> metadata".)
>
> These would make it far easier for browsers to run the up-to-date CSS
> test
> suite in their CI systems, to push fixes to the test suite, and to
> release
> tests their have into the test suite.
>
> /gsnedders

--
Test Format Guidelines
http://testthewebforward.org/docs/test-format-guidelines.html

Test Style Guidelines
http://testthewebforward.org/docs/test-style-guidelines.html

Test Templates
http://testthewebforward.org/docs/test-templates.html

CSS Naming Guidelines
http://testthewebforward.org/docs/css-naming.html

Test Review Checklist
http://testthewebforward.org/docs/review-checklist.html

CSS Metadata
http://testthewebforward.org/docs/css-metadata.html

Reply | Threaded
Open this post in threaded view
|

Re: Getting browser vendors running and submitting tests

James Graham-8
On 29/10/15 09:25, Gérard Talbot wrote:

> Le 2015-10-28 04:21, Geoffrey Sneddon a écrit :
>> [CC'd to public-test-infra; this is really mostly about CSS, can we avoid
>> cross-posting replies and keeping them on public-css-testsuite?]
>>
>> Myself and fantasai chaired a session at TPAC earlier today on this
>> subject. The minutes are at <
>> http://www.w3.org/2015/10/28-testing-minutes.html> (thanks fantasai!).
>>
>> Notably:
>>
>> * The build system makes it difficult for vendors to upstream their fixes
>> and their own tests (as vendors can't just fix the files they run; they
>> have to find some other file, fix that, and then work out how to build
>> the
>> test suite). As a result, they just fix the built files and don't
>> upstream
>> the fixes and often avoid updating the copy of the test suite because it
>> would trample their fixes.
>
> Why they don't fix their submitted tests (their tests in
> http://test.csswg.org/source/ ) and then re-import the modified test
> suite in their house system ?

They can, in theory, but won't, in practice. Typically developers have a
choice between writing tests that will be upstreamed and writing tests
that run in some single-browser system. They also have lots of pressure
to land new features to meet goals, keep their browser competitive, and
so on. Therefore they are likely to pick whatever testing system
produces the least friction. A system where you have to edit files in
one format then repull from upstream has way more friction than a system
where you just edit the file that you want to change and commit it to
your repository. Single browser test setups always have the latter
workflow. We need to replicate that for cross-browser tests if we want
developers to actually use them. With web-platform-tests we have managed
this with gecko and servo, and are starting to see much more developer
engagement with the web-platform-tests as a result.

Reply | Threaded
Open this post in threaded view
|

Re: Getting browser vendors running and submitting tests

fantasai
In reply to this post by Geoffrey Sneddon-4
On 10/28/2015 05:21 PM, Geoffrey Sneddon wrote:

> [CC'd to public-test-infra; this is really mostly about CSS, can we avoid cross-posting replies and keeping them on
> public-css-testsuite?]
>
> Myself and fantasai chaired a session at TPAC earlier today on this subject. The minutes are at
> <http://www.w3.org/2015/10/28-testing-minutes.html> (thanks fantasai!).
>
> Notably:
>
> * The build system makes it difficult for vendors to upstream their fixes and their own tests (as vendors can't just fix the
> files they run; they have to find some other file, fix that, and then work out how to build the test suite). As a result, they
> just fix the built files and don't upstream the fixes and often avoid updating the copy of the test suite because it would
> trample their fixes. We should get rid of the build system (at least insofar as it modifies/moves files), as I don't think
> anyone likely to run the tests doesn't support HTML at this point (yay the HTML5 parser!).
>
> * We should get rid of metadata, at least in the common case, required to submit tests to the the test suite, as this is an
> impediment to upstreaming tests. (See also my recent thread on public-css-testsuite, "Simplifying metadata".)

I disagree with this. I think we should reduce the metadata,
but there are some things (e.g. spec section associations)
that we need to keep.

~fantasai

Reply | Threaded
Open this post in threaded view
|

Re: Getting browser vendors running and submitting tests

James Graham-8
On 29/10/15 11:37, fantasai wrote:

> I disagree with this. I think we should reduce the metadata,
> but there are some things (e.g. spec section associations)
> that we need to keep.

FWIW the counterpoint to this is that people have and will point-blank
refuse to submit tests when there are requirements for metadata beyond
what is strictly needed to make the tests run.

I understand why this additional metadata is nice to have particularly
when you come back to tests later, but requiring it will cause people to
not upstream tests that they otherwise would have. I don't have a great
solution for you, but consider if they are ways to make more of the
metadata implicit in e.g. the directory structure, file naming, <title>
element, etc.

Reply | Threaded
Open this post in threaded view
|

Re: Getting browser vendors running and submitting tests

Chris Lilley
In reply to this post by Geoffrey Sneddon-4
Hello Geoffrey,

Wednesday, October 28, 2015, 5:21:33 PM, you wrote:
> * We should get rid of metadata, at least in the common case,

The crucial bit of metadata is a link to *what section of the spec is
actually tested*.

If someone writing a test cannot provide a link to  what is actually
being tested, the test is at best worthless and at worst, damaging.
But for the common case, a test author does know, they have the
relevant spec open as they write the test, so dropping in that link
is trivial to do.

Certainly there are other metadata items which are optional, but this
one is simple to provide and adds a lot of value.

--
Best regards,
 Chris  Lilley
 Technical Director, W3C Interaction Domain


Reply | Threaded
Open this post in threaded view
|

Re: Getting browser vendors running and submitting tests

Geoffrey Sneddon-4
On Thu, Oct 29, 2015 at 1:44 PM, Chris Lilley <[hidden email]> wrote:
Hello Geoffrey,

Wednesday, October 28, 2015, 5:21:33 PM, you wrote:
> * We should get rid of metadata, at least in the common case,

The crucial bit of metadata is a link to *what section of the spec is
actually tested*.

If someone writing a test cannot provide a link to  what is actually
being tested, the test is at best worthless and at worst, damaging.
But for the common case, a test author does know, they have the
relevant spec open as they write the test, so dropping in that link
is trivial to do.

Certainly there are other metadata items which are optional, but this
one is simple to provide and adds a lot of value.
 
The majority of tests browser vendors write do not have any sort of metadata in them—as James says, browser vendors will and have point-blank refused to submit tests that require any such metadata. If we want our test suite to actually be useful and contain the majority of tests browsers run, we need to get to a point where browser vendors are willing to submit tests. Even if you somehow convince browser vendors to include links in all future tests they write, this still leaves us without the tens of thousands of tests browser vendors already have.

As has been pointed out on numerous occasions (though I think many F2F rather in the summaries I've written here), we can make this metadata implicit in the directory structure, and that's something people *are* willing to work with (because it doesn't require modifying, quite possibly by hand, and reviewing, existing tests).

/Geoffrey
Reply | Threaded
Open this post in threaded view
|

Re: Getting browser vendors running and submitting tests

James Graham-8
In reply to this post by Chris Lilley
On 29/10/15 13:44, Chris Lilley wrote:

> Hello Geoffrey,
>
> Wednesday, October 28, 2015, 5:21:33 PM, you wrote:
>> * We should get rid of metadata, at least in the common case,
>
> The crucial bit of metadata is a link to *what section of the spec is
> actually tested*.
>
> If someone writing a test cannot provide a link to  what is actually
> being tested, the test is at best worthless and at worst, damaging.
> But for the common case, a test author does know, they have the
> relevant spec open as they write the test, so dropping in that link
> is trivial to do.
>
> Certainly there are other metadata items which are optional, but this
> one is simple to provide and adds a lot of value.
>

So I agree that sounds eminently reasonable. OTOH a random trawl through
reftests in mozilla-central turned up very few where people had added
this kind of link. So I conclude that even if it sounds like it's a good
idea it isn't an emergent behaviour of browser developers and
constitutes an extra burden for upstreaming tests compared to writing
browser-specific tests.

Reply | Threaded
Open this post in threaded view
|

Re: Getting browser vendors running and submitting tests

Gérard Talbot-3
In reply to this post by James Graham-8
Le 2015-10-28 22:48, James Graham a écrit :
> On 29/10/15 11:37, fantasai wrote:
>
>> I disagree with this. I think we should reduce the metadata,
>> but there are some things (e.g. spec section associations)
>> that we need to keep.
>
> FWIW the counterpoint to this is that people have and will point-blank
> refuse to submit tests when there are requirements for metadata beyond
> what is strictly needed to make the tests run.

<link rel="author" href="...">: very easy to do


<meta name="flags" content="...">: is that really difficult to edit? I
don't think so.



<title> text: the documentation should help by giving useful examples.
James Hopkins had a good and easy system to use. He was just putting the
most important css property and then another one which was being tested.
You can not be wrong 99% of the time if you do this systematically.

<title>CSS [Module name] Test: property name - property name</title>

eg

<title>CSS Writing Modes Test: vertical-align - 'super' and vertical-rl
writing-mode

<link rel="help" href="...">: it should be easy if you use the James
Hopkins system. Just link to both properties you use in the <title>.


<meta name="assert" content="...">: that one is more difficult. But
then, all reviewers need is some useful, meaningful, helpful comments at
judicious spots and/or useful, meaningful, semantic id attribute, class
attribute, function identifiers.

Examples:

(bad) class="class1"
(bad) class="class2"
(bad) class="one"
(bad) class="two"
(bad) class="a"
(bad) class="b"
(bad) id="div1"
(bad) id="div2"
(bad) id="span1"
(bad) id="span2"
(bad) id="one"
(bad) id="two"
(bad) id="a"
(bad) id="b"
(bad) function test()

Better, more recommendable:

(good) class="abs-pos-children"
(good) id="rel-pos-grand-parent"
(good) id="containing-block"
(good) id="blue-float-left"
(good) id="red-overlapped"
(good) id="green-overlapping"
(good) id="reference"
(good) id="control"
(good) id="clearing"
(good) id="following-sibling"
(good) class="adjacent-sibling"
(good) class="invalid"

Good identifiers are those which are
- descriptive with regards to the working/building logic of the test,
- descriptive of the design logic of the test or
- descriptive of the respective position in the containment hierarchy or
- descriptive of the respective position in the positioning hierarchy.


> I understand why this additional metadata is nice to have particularly
> when you come back to tests later, but requiring it will cause people
> to not upstream tests that they otherwise would have. I don't have a
> great solution for you, but consider if they are ways to make more of
> the metadata implicit in e.g. the directory structure, file naming,
> <title> element, etc.


The documentation could be more helpful.
http://testthewebforward.org/docs/test-templates.html
does not provide real examples.

I'm sure the documentation could be improved...

Gérard
--
Test Format Guidelines
http://testthewebforward.org/docs/test-format-guidelines.html

Test Style Guidelines
http://testthewebforward.org/docs/test-style-guidelines.html

Test Templates
http://testthewebforward.org/docs/test-templates.html

CSS Naming Guidelines
http://testthewebforward.org/docs/css-naming.html

Test Review Checklist
http://testthewebforward.org/docs/review-checklist.html

CSS Metadata
http://testthewebforward.org/docs/css-metadata.html

Reply | Threaded
Open this post in threaded view
|

Re: Getting browser vendors running and submitting tests

Florian Rivoal-4
In reply to this post by James Graham-8

> On 29 Oct 2015, at 11:48, James Graham <[hidden email]> wrote:
>
> On 29/10/15 11:37, fantasai wrote:
>
>> I disagree with this. I think we should reduce the metadata,
>> but there are some things (e.g. spec section associations)
>> that we need to keep.
>
> FWIW the counterpoint to this is that people have and will point-blank refuse to submit tests when there are requirements for metadata beyond what is strictly needed to make the tests run.
>
> I understand why this additional metadata is nice to have particularly when you come back to tests later, but requiring it will cause people to not upstream tests that they otherwise would have. I don't have a great solution for you, but consider if they are ways to make more of the metadata implicit in e.g. the directory structure, file naming, <title> element, etc.

I think that a test that has neither a pointer to the spec section it is testing nor an explicit assertion is close to being unusable, unreviewable, and unmaintainable, and that we don't loose much if it is not being submitted.

We should reduce the amount of metadata required, but not to 0.

 - Florian
Reply | Threaded
Open this post in threaded view
|

Re: Getting browser vendors running and submitting tests

Geoffrey Sneddon-4


On Thu, Oct 29, 2015 at 4:23 PM, Florian Rivoal <[hidden email]> wrote:

> On 29 Oct 2015, at 11:48, James Graham <[hidden email]> wrote:
>
> On 29/10/15 11:37, fantasai wrote:
>
>> I disagree with this. I think we should reduce the metadata,
>> but there are some things (e.g. spec section associations)
>> that we need to keep.
>
> FWIW the counterpoint to this is that people have and will point-blank refuse to submit tests when there are requirements for metadata beyond what is strictly needed to make the tests run.
>
> I understand why this additional metadata is nice to have particularly when you come back to tests later, but requiring it will cause people to not upstream tests that they otherwise would have. I don't have a great solution for you, but consider if they are ways to make more of the metadata implicit in e.g. the directory structure, file naming, <title> element, etc.

I think that a test that has neither a pointer to the spec section it is testing nor an explicit assertion is close to being unusable, unreviewable, and unmaintainable, and that we don't loose much if it is not being submitted.

We should reduce the amount of metadata required, but not to 0.

That implies that many tests that browsers have are "unusable, unreviewable, and unmaintainable", so I don't think that's a true statement! After all, it seems likely that browsers would've changed what they're doing if it were so bad.

/g

Reply | Threaded
Open this post in threaded view
|

Re: Getting browser vendors running and submitting tests

Florian Rivoal-4

On 29 Oct 2015, at 16:57, Geoffrey Sneddon <[hidden email]> wrote:



On Thu, Oct 29, 2015 at 4:23 PM, Florian Rivoal <[hidden email]> wrote:

> On 29 Oct 2015, at 11:48, James Graham <[hidden email]> wrote:
>
> On 29/10/15 11:37, fantasai wrote:
>
>> I disagree with this. I think we should reduce the metadata,
>> but there are some things (e.g. spec section associations)
>> that we need to keep.
>
> FWIW the counterpoint to this is that people have and will point-blank refuse to submit tests when there are requirements for metadata beyond what is strictly needed to make the tests run.
>
> I understand why this additional metadata is nice to have particularly when you come back to tests later, but requiring it will cause people to not upstream tests that they otherwise would have. I don't have a great solution for you, but consider if they are ways to make more of the metadata implicit in e.g. the directory structure, file naming, <title> element, etc.

I think that a test that has neither a pointer to the spec section it is testing nor an explicit assertion is close to being unusable, unreviewable, and unmaintainable, and that we don't loose much if it is not being submitted.

We should reduce the amount of metadata required, but not to 0.

That implies that many tests that browsers have are "unusable, unreviewable, and unmaintainable", so I don't think that's a true statement! After all, it seems likely that browsers would've changed what they're doing if it were so bad.

Let me qualify that statement a little. "unusable, unreviewable, and unmaintainable" to a third party.

When a browser vendor writes a test together with new implementation, or targeting an existing part of their implementation, they have this metadata (not the whole set of what we require today, but at least the "what are you testing" information), at least in their heads, probably in their bug tracker as well.

That makes the test useful to them, since they know what they are testing.

I challenge the idea that there are tests in the browsers vendors's private respositories that have been checked in independently of the code (not just in a separate commit, but on a branch that makes it impossible to know what code they relate to), and have no information anywhere (not in the commit message, not in the test, not in the bug tracker) about what these tests are.

If we allow tests to be shared entirely without information, this is the situation we would be in. Other browser vendors, receiving such a test, would suddenly have a  failing test (or a few hundred) in the CI test suite, with no indication about that that is about.

The overhead of figuring out what such a test does is far higher than the overhead of writing it down when you already know it. It is quite likely that, for the receiving party, such tests would indeed be "unusable, unreviewable, and unmaintainable".

 - Florian
Reply | Threaded
Open this post in threaded view
|

Re: Getting browser vendors running and submitting tests

Boris Zbarsky
In reply to this post by Chris Lilley
On 10/29/15 12:44 AM, Chris Lilley wrote:
> The crucial bit of metadata is a link to *what section of the spec is
> actually tested*.

For a large proportion of tests, what's being tested is the interaction
of multiple spec sections, if not multiple specs.

> Certainly there are other metadata items which are optional, but this
> one is simple to provide

I disagree with it being simple to provide.  The last time I tried this,
pretty much any test that was at all interesting needed half a dozen or
more links listed to provide this information.  I gave up on putting it
in tests.

-Boris


Reply | Threaded
Open this post in threaded view
|

Re: Getting browser vendors running and submitting tests

Boris Zbarsky
On 10/29/15 9:05 AM, Boris Zbarsky wrote:
> For a large proportion of tests, what's being tested is the interaction
> of multiple spec sections, if not multiple specs.

Just to "quantify" this...  Say you have N "spec sections" in the sense
of testable normative requirements.  That's N single-section tests.  If
any time a normative requirement applies only N/M of the remaining ones
do, that's N^2/M two-section tests.  In practice, N is a pretty large
number (in the hundreds) and M is somewhat smaller than N, so if you're
doing a decent job of writing tests you end up with a lot more tests
testing interaction than testing a single normative statement.

-Boris

Reply | Threaded
Open this post in threaded view
|

Re: Getting browser vendors running and submitting tests

Gérard Talbot-3
In reply to this post by Boris Zbarsky
Le 2015-10-29 09:05, Boris Zbarsky a écrit :
> On 10/29/15 12:44 AM, Chris Lilley wrote:
>> The crucial bit of metadata is a link to *what section of the spec is
>> actually tested*.
>
> For a large proportion of tests, what's being tested is the
> interaction of multiple spec sections, if not multiple specs.

I agree.

>> Certainly there are other metadata items which are optional, but this
>> one is simple to provide
>
> I disagree with it being simple to provide.  The last time I tried
> this, pretty much any test that was at all interesting needed half a
> dozen or more links listed to provide this information.

Boris, I have never created a test that had (or needed) more than 3
links to spec.

When a real webpage causes a bug report to be created, it is rare that a
reduced test (reproducing the problem) coming from the real webpage
involves the interaction of more than 3 spec sections.

Gérard

>  I gave up on
> putting it in tests.
>
> -Boris



--
Test Format Guidelines
http://testthewebforward.org/docs/test-format-guidelines.html

Test Style Guidelines
http://testthewebforward.org/docs/test-style-guidelines.html

Test Templates
http://testthewebforward.org/docs/test-templates.html

CSS Naming Guidelines
http://testthewebforward.org/docs/css-naming.html

Test Review Checklist
http://testthewebforward.org/docs/review-checklist.html

CSS Metadata
http://testthewebforward.org/docs/css-metadata.html