Testsuite flags

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Testsuite flags

Geoffrey Sneddon-4
(Bcc'd public-css-testsuite; can we please keep responses on a single
mailing list, in this case www-style as it's about a WG resolution?)

At the F2F in May, we decided to do this on the list, so, uh, here goes:

We currently have a whole load of flags, documented over at
<http://testthewebforward.org/docs/css-metadata.html>:

* ahem
* animated
* asis
* combo
* dom
* font
* history
* HTMLonly
* http
* image
* interact
* invalid
* may
* namespace
* nonHTML
* paged
* scroll
* should
* speech
* svg
* userstyle
* 32bit
* 96dpi

I believe we have rough consensus to drop:

* ahem
* http
* image
* namespace
* svg
* 32bit
* 96dpi

ahem is something we should just rely on as an assumption for the whole
testsuite (this is the status-quo for web-platform-tests)

http is redundant as we move to .headers files and away from .htaccess,
as you can trivially statically determine the property

image, namespace, and svg are all catering for theoretical UAs that
don't implement the respective features.

32bit is similar, insofar as nobody is aware of any current UA that
doesn't use 32bit integers everywhere.

96dpi has been entirely redundant since CSS 2.1 defined 1 CSS inch to
equal 96px.

In addition to this, once we drop the build system, asis, HTMLonly, and
nonHTML can be nuked because they become meaningless, as they exist only
to control it.

This leaves us with:

* animated
* combo
* dom
* font
* history
* interact
* invalid
* may
* paged
* scroll
* should
* speech
* userstyle

Now, without rough consensus:

I think we should drop combo. We added it for the sake of CR-exit
criteria and because we had some tests that were just the sum of others.
We probably can use case-by-case judgement better than we can the flag
to do anything here.

I think we can drop dom, because script can be statically detected
easily enough (i.e., is there a <script> element?). I think Florian had
some edge-case he was concerned with around here? (I want to say that
was as far back as TPAC last year he mentioned that to me!)

I'd like to drop font, and vastly reduce the number of tests that
require fonts to be installed by just relying on @font-face more (I hope
we've reached a point where we can rely on it now!). Obviously we can't
completely eliminate having to install fonts, but we can make it rare.

I'm also going to point out that as we try and converge on
web-platform-tests policies we're going to end up requiring "-manual"
filename suffixes on all tests flagged with animated, font, history,
interact, paged, speech, or userstyle. We may want to drop some of these
prior to that, however (or rather, more likely, say "these are
deprecated and should be treated identically to interact").

/gsnedders

Reply | Threaded
Open this post in threaded view
|

Re: Testsuite flags

fantasai
On 08/17/2016 09:48 PM, Geoffrey Sneddon wrote:
>
> I think we should drop combo. We added it for the sake of CR-exit
> criteria and because we had some tests that were just the sum of others.
> We probably can use case-by-case judgement better than we can the flag
> to do anything here.

I think this is fair.

> I think we can drop dom, because script can be statically detected
> easily enough (i.e., is there a <script> element?). I think Florian had
> some edge-case he was concerned with around here? (I want to say that
> was as far back as TPAC last year he mentioned that to me!)

The case was, IIRC, a test which is easier to perform without script
but has some scripting as an aid.

> I'd like to drop font, and vastly reduce the number of tests that
> require fonts to be installed by just relying on @font-face more (I hope
> we've reached a point where we can rely on it now!). Obviously we can't
> completely eliminate having to install fonts, but we can make it rare.

I think we can, indeed, do that. If a UA doesn't support @font-face,
then it can just assume that any test that includes @font-face and
isn't testing @font-face requires special font installation.
We should document that maybe. :)

> I'm also going to point out that as we try and converge on
> web-platform-tests policies we're going to end up requiring "-manual"
> filename suffixes on all tests flagged with animated, font, history,
> interact, paged, speech, or userstyle. We may want to drop some of these
> prior to that, however (or rather, more likely, say "these are
> deprecated and should be treated identically to interact").

I think it's reasonably likely that some UA will have an automated way
to handle animated tests at some point. So it seems reasonable to keep
that separate.

Paged tests can be automated in Gecko. The are not manual.

Speech would be good to distinguish. They're likely to be handled
in a separate run from everything else.

Userstyle, history, and interact can maybe be collapsed. Though IIRC
the first two can be automated, just not without proprietary testing
infrastructure--and for that reason we may want them kept distinct.

~fantasai

Reply | Threaded
Open this post in threaded view
|

Re: Testsuite flags

Gérard Talbot-2
In reply to this post by Geoffrey Sneddon-4
Le 2016-08-18 00:48, Geoffrey Sneddon a écrit :

> (Bcc'd public-css-testsuite; can we please keep responses on a single
> mailing list, in this case www-style as it's about a WG resolution?)
>
> At the F2F in May, we decided to do this on the list, so, uh, here
> goes:
>
> We currently have a whole load of flags, documented over at
> <http://testthewebforward.org/docs/css-metadata.html>:
>
> * ahem
> * animated
> * asis

'asis' flag is supposed prevents reserialization. If the source code,
for example, uses &#xFF12;&#xFF18; (full width characters), then I
should see, read character entity in the built tests... but it does not;
we get 28 instead of 28 (note how difficult it can be to distinguish
non-full width characters from full width characters; therefore the use
of character entities).

Eg
[src, test]
http://test.csswg.org/source/css-writing-modes-3/full-width-002.html

[built, test]
http://test.csswg.org/suites/css-writing-modes-3_dev/nightly-unstable/html/full-width-002.htm


> * combo
> * dom
> * font
> * history

'history' flag was mostly to be used for testing :link versus :visited
links.

> * HTMLonly
> * http
> * image
> * interact
> * invalid
> * may
> * namespace
> * nonHTML
> * paged
> * scroll
> * should
> * speech
> * svg
> * userstyle
> * 32bit
> * 96dpi
>
> I believe we have rough consensus to drop:
>
> * ahem
> * http
> * image
> * namespace
> * svg
> * 32bit
> * 96dpi
>
> ahem is something we should just rely on as an assumption for the whole
> testsuite (this is the status-quo for web-platform-tests)
>
> http is redundant as we move to .headers files and away from .htaccess,
> as you can trivially statically determine the property
>
> image, namespace, and svg are all catering for theoretical UAs that
> don't implement the respective features.
>
> 32bit is similar, insofar as nobody is aware of any current UA that
> doesn't use 32bit integers everywhere.

Konqueror, Opera 12.17 and possibly a few other UAs do not support 32bit
z-index values; that's why the 32bit flag was introduced. But I do not
really care; we can drop 32bit flag.

>
> 96dpi has been entirely redundant since CSS 2.1 defined 1 CSS inch to
> equal 96px.
>
> In addition to this, once we drop the build system, asis, HTMLonly, and
> nonHTML can be nuked because they become meaningless, as they exist
> only
> to control it.
>
> This leaves us with:
>
> * animated
> * combo
> * dom
> * font
> * history
> * interact
> * invalid
> * may
> * paged
> * scroll
> * should
> * speech
> * userstyle
>
> Now, without rough consensus:
>
> I think we should drop combo. We added it for the sake of CR-exit
> criteria and because we had some tests that were just the sum of
> others.
> We probably can use case-by-case judgement better than we can the flag
> to do anything here.
>
> I think we can drop dom, because script can be statically detected
> easily enough (i.e., is there a <script> element?). I think Florian had
> some edge-case he was concerned with around here? (I want to say that
> was as far back as TPAC last year he mentioned that to me!)
>
> I'd like to drop font, and vastly reduce the number of tests that
> require fonts to be installed by just relying on @font-face more (I
> hope
> we've reached a point where we can rely on it now!). Obviously we can't
> completely eliminate having to install fonts, but we can make it rare.

I would agree but we could offer to use some specific fonts. Eg.
NotoSansDeseret-Regular.ttf
which is needed in some tests. I'm sorry... I do not have time to
elaborate right now on this and other points of your email ...


Gérard (in a hurry)

>
> I'm also going to point out that as we try and converge on
> web-platform-tests policies we're going to end up requiring "-manual"
> filename suffixes on all tests flagged with animated, font, history,
> interact, paged, speech, or userstyle. We may want to drop some of
> these
> prior to that, however (or rather, more likely, say "these are
> deprecated and should be treated identically to interact").
>
> /gsnedders

Reply | Threaded
Open this post in threaded view
|

Re: Testsuite flags

Florian Rivoal-4
In reply to this post by Geoffrey Sneddon-4

> On Aug 18, 2016, at 06:48, Geoffrey Sneddon <[hidden email]> wrote:
>
> I believe we have rough consensus to drop:
>
> * ahem
> * http
> * image
> * namespace
> * svg
> * 32bit
> * 96dpi
>

Agreed.

> Now, without rough consensus:
>
> I think we should drop combo. We added it for the sake of CR-exit
> criteria and because we had some tests that were just the sum of others.
> We probably can use case-by-case judgement better than we can the flag
> to do anything here.

Agreed.

> I think we can drop dom, because script can be statically detected
> easily enough (i.e., is there a <script> element?). I think Florian had
> some edge-case he was concerned with around here? (I want to say that
> was as far back as TPAC last year he mentioned that to me!)

More of a curiosity than a concern. It is possible to use scripts in a test to make it easier to run (automate part of it to reduce the number of manual steps) while still keeping the test valid and usable if there is no support for script. This is a valid case for using scripts but not applying the dom flag. However, that's really a corner case of limited significance, so I'm OK with dropping the dom flag and inferring it from the presence of script. The false positives will be few and far between, and of little consequence.

> I'd like to drop font, and vastly reduce the number of tests that
> require fonts to be installed by just relying on @font-face more (I hope
> we've reached a point where we can rely on it now!). Obviously we can't
> completely eliminate having to install fonts, but we can make it rare.
>
> I'm also going to point out that as we try and converge on
> web-platform-tests policies we're going to end up requiring "-manual"
> filename suffixes on all tests flagged with animated, font, history,
> interact, paged, speech, or userstyle. We may want to drop some of these
> prior to that, however (or rather, more likely, say "these are
> deprecated and should be treated identically to interact").

I am not sure that speech and paged fall into the same category as the rest.

I agree with merging all the other types of "can't automate fully" (animated, font, history, interact, userstyle) flags into a single category. We should continue to recognize these flags going forward for legacy reasons, but we can treat them as equivalent. Going forward, we can use the "interact" flag, or go with using a -manual suffix in the file name, or allow either. I don't care strongly.

 - Florian
Reply | Threaded
Open this post in threaded view
|

Re: Testsuite flags

Gérard Talbot-2
In reply to this post by fantasai
Le 2016-08-18 02:15, fantasai a écrit :
> On 08/17/2016 09:48 PM, Geoffrey Sneddon wrote:
>>

>> I'd like to drop font, and vastly reduce the number of tests that
>> require fonts to be installed by just relying on @font-face more (I
>> hope
>> we've reached a point where we can rely on it now!). Obviously we
>> can't
>> completely eliminate having to install fonts, but we can make it rare.
>
> I think we can, indeed, do that. If a UA doesn't support @font-face,
> then it can just assume that any test that includes @font-face and
> isn't testing @font-face requires special font installation.
> We should document that maybe. :)



Regarding font flag
- - - - - - - - - -

I am for dropping the font flag and use @font-face, but...

How do you propose that we should use the @font-face then? With or
without woff files? I ask because .woff files and .woff2 files may not
be rendered the same way across browsers... How well do we know if woff
files are rendered the same across browsers and platforms?

Other questions.

Shouldn't we always use src: local() to avoid, prevent downloading and
loading processes?

If the font file to be downloaded and installed (.otf or .ttf and/or
.woff or .woff2)'s filesize is big (say > 50Kb), can this affect
automation of test results? Eg mplus-1p-regular.woff filesize is greater
than 700Kb and it is currently being used in 468 tests right now.

Let's say I copy and paste the mplus-1p-regular.woff and the
mplus-1p-regular.ttf files in a /support folder. Then what would be the
best recommendable coding practice for testing and test authors here?

  @font-face
    {
      font-family: "mplus-1p-regular";
      src: url("support/mplus-1p-regular.woff") format("woff");
      /* filesize: 803300 bytes (784.5 KBytes) */
    }

or

  @font-face
    {
      font-family: "mplus-1p-regular";
      src: local(M+1p) ,
      url(support/mplus-1p-regular.ttf) format("opentype") ,
      url("support/mplus-1p-regular.woff") format("woff");
    }

or something else?

I am convinced we need some clear, explicit guidelines on how to use
@font-face.


>> I'm also going to point out that as we try and converge on
>> web-platform-tests policies we're going to end up requiring "-manual"
>> filename suffixes on all tests flagged with animated, font, history,
>> interact, paged, speech, or userstyle. We may want to drop some of
>> these
>> prior to that, however (or rather, more likely, say "these are
>> deprecated and should be treated identically to interact").
>
> I think it's reasonably likely that some UA will have an automated way
> to handle animated tests at some point. So it seems reasonable to keep
> that separate.


Regarding animated, interact and scroll flags
- - - - - - - - - - - - - - - - - - - - - - -

Several HTML-to-PDF web-aware softwares (eg: Prince 10 release 7,
AntennaHouse Formatter V6 [2] and possibly others that I am not aware
of) are able to support a great deal of CSS. Those 3 flags help refine
their test suite results.

[1]: Prince version 10, CSS Properties
https://www.princexml.com/doc/properties/

[2]: AntennaHouse Formatter V6.3's CSS implementation
https://www.antennahouse.com/product/ahf63/ahf-css6.html


> Paged tests can be automated in Gecko. The are not manual.

They require some code too:
<html class="reftest-print">

https://bugzilla.mozilla.org/show_bug.cgi?id=685012#c49
See comments 49 to 57 for a discussion on this.

Gérard