Test assertions requirement

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Test assertions requirement

Geoffrey Sneddon-4
(Bcc'd public-css-testsuite; can we please keep responses on a single
mailing list, in this case www-style as it's about a WG resolution?)

In the SF F2F, 2016-05-09 we resolved:

RESOLVED: testharness.js tests don't need a meta assert (but reftests
still do)

>From memory, this was based on a mistaken understanding that assertions
were currently required everywhere.

This seems to contradict all documentation going back almost a decade!
In 2007 fantasai updated the wiki in
<https://wiki.csswg.org/test/format?do=diff&rev2%5B0%5D=1184350761&rev2%5B1%5D=1186099916&difftype=sidebyside>
to state that assertions are optional (generically!).

I've been unable to find any discussion leading to this change, though
obviously discussion was happening around then (see
<https://lists.w3.org/Archives/Public/public-css-testsuite/2007Jun/0024.html>
for example).

Do we want to revisit the resolution in light of being mistaken as to
previous policy? Or do we just want to consider the documentation wrong?

/Geoffrey

Reply | Threaded
Open this post in threaded view
|

Re: Test assertions requirement

Florian Rivoal-4

> On Aug 16, 2016, at 00:18, Geoffrey Sneddon <[hidden email]> wrote:
>
> (Bcc'd public-css-testsuite; can we please keep responses on a single
> mailing list, in this case www-style as it's about a WG resolution?)
>
> In the SF F2F, 2016-05-09 we resolved:
>
> RESOLVED: testharness.js tests don't need a meta assert (but reftests
> still do)
>
>> From memory, this was based on a mistaken understanding that assertions
> were currently required everywhere.
>
> This seems to contradict all documentation going back almost a decade!
> In 2007 fantasai updated the wiki in
> <https://wiki.csswg.org/test/format?do=diff&rev2%5B0%5D=1184350761&rev2%5B1%5D=1186099916&difftype=sidebyside>
> to state that assertions are optional (generically!).
>
> I've been unable to find any discussion leading to this change, though
> obviously discussion was happening around then (see
> <https://lists.w3.org/Archives/Public/public-css-testsuite/2007Jun/0024.html>
> for example).
>
> Do we want to revisit the resolution in light of being mistaken as to
> previous policy? Or do we just want to consider the documentation wrong?


The documentation here as well says it is optional:
http://testthewebforward.org/docs/css-metadata.html

Regardless of the previous status, I'd rather consider keep the requirement for the meta assert in reftests, but if others insists we drop it (back?) from must to should, I guess I can begrudgingly agree...

 - Florian
Reply | Threaded
Open this post in threaded view
|

Re: Test assertions requirement

r12a
On 16/08/2016 07:57, Florian Rivoal wrote:
> if others insists we drop it (back?) from must to should, I guess I can begrudgingly agree...

i don't think i would.  I thought i recently read that a decision was
taken to keep keep assertions, but i can't remember where.  Maybe PLH knows?

the optional in the documentation dates, i think, from the beginning,
and therefore substantially predates any recent discussions.  I wouldn't
take it prima facie as evidence that a decision was taken to make
assertions optional.

ri

Reply | Threaded
Open this post in threaded view
|

Re: Test assertions requirement

Gérard Talbot-2
In reply to this post by Geoffrey Sneddon-4
Le 2016-08-15 18:18, Geoffrey Sneddon a écrit :

> (Bcc'd public-css-testsuite; can we please keep responses on a single
> mailing list, in this case www-style as it's about a WG resolution?)
>
> In the SF F2F, 2016-05-09 we resolved:
>
> RESOLVED: testharness.js tests don't need a meta assert (but reftests
> still do)
>
>> From memory, this was based on a mistaken understanding that
>> assertions
> were currently required everywhere.
>
> This seems to contradict all documentation going back almost a decade!
> In 2007 fantasai updated the wiki in
> <https://wiki.csswg.org/test/format?do=diff&rev2%5B0%5D=1184350761&rev2%5B1%5D=1186099916&difftype=sidebyside>
> to state that assertions are optional (generically!).

First, I was not in the SF F2F event.

My understanding is: if a test is simple and basic in such a way that
glancing at the source code makes easy to figure out, to understand the
purpose/goal of such test, then the test assertion text is not needed.
But this is an exception rather than a rule. A clear majority of CSS
tests are complex, not-obvious-to-figure-out, are testing a specific
statement of the specification or module (or should be in the first
place about testing a specific statement of a specification or module or
claiming to be testing some specific statement). Therefore test assert
text are often needed, desirable, suitable. And with the multiplication
of specialized modules, this trend will not diminish with time.



> I've been unable to find any discussion leading to this change, though
> obviously discussion was happening around then (see
> <https://lists.w3.org/Archives/Public/public-css-testsuite/2007Jun/0024.html>
> for example).


I swear I read discussions about test assert text in this mailing list
before and not too long ago ...

Gérard

> Do we want to revisit the resolution in light of being mistaken as to
> previous policy? Or do we just want to consider the documentation
> wrong?
>
> /Geoffrey



Reply | Threaded
Open this post in threaded view
|

Re: Test assertions requirement

fantasai
In reply to this post by Geoffrey Sneddon-4
On 08/15/2016 03:18 PM, Geoffrey Sneddon wrote:

> (Bcc'd public-css-testsuite; can we please keep responses on a single
> mailing list, in this case www-style as it's about a WG resolution?)
>
> In the SF F2F, 2016-05-09 we resolved:
>
> RESOLVED: testharness.js tests don't need a meta assert (but reftests
> still do)
>
>>From memory, this was based on a mistaken understanding that assertions
> were currently required everywhere.
>
> This seems to contradict all documentation going back almost a decade!
> In 2007 fantasai updated the wiki in
> <https://wiki.csswg.org/test/format?do=diff&rev2%5B0%5D=1184350761&rev2%5B1%5D=1186099916&difftype=sidebyside>
> to state that assertions are optional (generically!).
>
> I've been unable to find any discussion leading to this change, though
> obviously discussion was happening around then (see
> <https://lists.w3.org/Archives/Public/public-css-testsuite/2007Jun/0024.html>
> for example).
>
> Do we want to revisit the resolution in light of being mistaken as to
> previous policy? Or do we just want to consider the documentation wrong?

I think this is just unclear wording in the resolution. The resolution
meant that we still need a standardized way of expressing the assertions
in reftests -- that it's still considered part of the metadata we maintain
for the test suite. It was not intended to make the assertion required.

That said I agree with gtalbot and r12a that we should *recommend* people
document the purpose of their test in its header in the same way we
*recommend* people document the purpose of their function in their code.
Sadly the quality of test asserts is absymal usually. :( But good asserts,
like good docstrings, should be encouraged...

~fantasai