Re: Towards a better testsuite: Metadata

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

fantasai
On 03/24/2016 01:00 PM, Geoffrey Sneddon wrote:

>
> I went through all of the all of the metadata in
> <https://lists.w3.org/Archives/Public/public-css-testsuite/2015Oct/0004.html>;
> there's rough agreement on removing much of what we currently have.
> That said, I think it's worthwhile to reiterate that requiring *any*
> metadata causes friction. Tests written by browser vendors are rarely
> a file or two which is quick to add metadata too. I know in general
> people seem interested in using the same infrastructure to run both
> web-platform-tests and csswg-test, which essentially requires the
> metadata required to run the tests be identical across the two.
>
> ...
>
> To outline what I'd like to see happen:
>
> - Get rid of the build system, replacing many of it's old errors with
> a lint tool that tests for them.
> - Policy changes to get rid of all metadata in the common case.
> - Change the commit policy. (Require review first, require no new lint errors.)

As I wrote in
   https://www.w3.org/mid/5702DA58.1020703@...
I agree with Florian's comments about metadata, and disagree
strongly with simply doing away with all of it. But I think
we can simplify things down to:

<!DOCTYPE html>
<title>Explain *exactly* what situation you're testing
and what it's supposed to do</title>
<link rel="help" href="spec">
<link rel="match" href="reference">

plus optional <meta name=flags> if needed for that test.
(I agree with trimming down the flags as well; many are
not really necessary at this point.)

This is short enough and straightforward enough that I think
anyone can learn to write a new test off the top of their
head after having written two or three. We can also have a
command line tool that generates the template, which is
something Mozilla already has built into its mach system for
writing WPT.


Wrt filenames, I would like us to recommend
   a) indexes be zero-filled to three places so tests sort correctly
   b) hyphens over underscore
but don't think we need to actually require anything here,
other than that it be descriptive of its content and not
obnoxiously long.

~fantasai

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

Florian Rivoal-4

> On Apr 9, 2016, at 02:00, fantasai <[hidden email]> wrote:
>
> I think we can simplify things down to:
>
> <!DOCTYPE html>
> <title>Explain *exactly* what situation you're testing
> and what it's supposed to do</title>
> <link rel="help" href="spec">
> <link rel="match" href="reference">
>
> plus optional <meta name=flags> if needed for that test.
> (I agree with trimming down the flags as well; many are
> not really necessary at this point.)

Works for me. The only point I am not 100% sure about is putting the assertion/explanation in the title, as I think people have expectations that a title should be a few words, rather than one or two full sentences, and I worry a bit that this set up will give us under-described tests.

I think I'd prefer keeping the assertion in an assert meta, and effectively disregarding what goes into the title, but I don't know what's easier to teach and enforce:
- Writing long descriptive titles
- Having an assert meta

But either way, this is the right amount of information. We can discuss a little bit about whether this is the ideal way to mark it up or not, but if that's that format that people will accept I'm fine with it.

 - Florian
Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

r12a
On 11/04/2016 01:56, Florian Rivoal wrote:
> Works for me. The only point I am not 100% sure about is putting the assertion/explanation in the title, as I think people have expectations that a title should be a few words, rather than one or two full sentences, and I worry a bit that this set up will give us under-described tests.

I very much agree. I think that assertions in the title will become a
very unwieldy in many cases (eg. where you have to say "If
such-and-such, then so-and-so will happen."), and lead to unhelpful
brevity in others.

For me the assert field is a top priority.

I find the titles useful to give a quick idea of the intent of the test
(see for example the distinction at
https://www.w3.org/International/tests/repo/results/the-lang-attribute).
  But i wonder whether we can make the title optional - if needed, but
missing, it may be possible to use the file name as the title.

In any case, i'm hoping that we won't stop people putting in more
metadata if they really want to.

ri

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

fantasai
On 04/11/2016 03:55 AM, [hidden email] wrote:

> On 11/04/2016 01:56, Florian Rivoal wrote:
>> Works for me. The only point I am not 100% sure about is putting the assertion/explanation in the title, as I think people
>> have expectations that a title should be a few words, rather than one or two full sentences, and I worry a bit that this set
>> up will give us under-described tests.
>
> I very much agree. I think that assertions in the title will become a very unwieldy in many cases (eg. where you have to say
> "If such-and-such, then so-and-so will happen."), and lead to unhelpful brevity in others.
>
> For me the assert field is a top priority.
>
> I find the titles useful to give a quick idea of the intent of the test (see for example the distinction at
> https://www.w3.org/International/tests/repo/results/the-lang-attribute).  But i wonder whether we can make the title optional
> - if needed, but missing, it may be possible to use the file name as the title.

That was my intention when we drew up the metadata documentation,
but it seems that a lot of people find the metadata confusing to
write and are annoyed at the amount of markup it requires. And
I can't say I blame them; there's enough that even I can't write
a new test off the top of my head if I haven't looked things up
recently enough.

<title> tags are structurally simple. They're also required for
the HTML to be valid, so we have to have one anyway.
   https://html.spec.whatwg.org/multipage/semantics.html#the-head-element

If we'd like to make a distinction between a "title" and a longer
"description", then maybe we can adopt a convention similar to
commit messages: the first line, if it stands alone, is the title,
and the rest is the description.

I'd rather keep the metadata and its markup as simple as possible.
It should be easy to write a new test from a blank page once you've
written 2-3. It should therefore require as little typing and
memorization as possible; and <meta> tags don't facilitate this at
all because they're noisy.

~fantasai

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

Florian Rivoal-4

> On Apr 12, 2016, at 02:56, fantasai <[hidden email]> wrote:
>
> On 04/11/2016 03:55 AM, [hidden email] wrote:
>> On 11/04/2016 01:56, Florian Rivoal wrote:
>>> Works for me. The only point I am not 100% sure about is putting the assertion/explanation in the title, as I think people
>>> have expectations that a title should be a few words, rather than one or two full sentences, and I worry a bit that this set
>>> up will give us under-described tests.
>>
>> I very much agree. I think that assertions in the title will become a very unwieldy in many cases (eg. where you have to say
>> "If such-and-such, then so-and-so will happen."), and lead to unhelpful brevity in others.
>>
>> For me the assert field is a top priority.
>>
>> I find the titles useful to give a quick idea of the intent of the test (see for example the distinction at
>> https://www.w3.org/International/tests/repo/results/the-lang-attribute).  But i wonder whether we can make the title optional
>> - if needed, but missing, it may be possible to use the file name as the title.
>
> That was my intention when we drew up the metadata documentation,
> but it seems that a lot of people find the metadata confusing to
> write and are annoyed at the amount of markup it requires. And
> I can't say I blame them; there's enough that even I can't write
> a new test off the top of my head if I haven't looked things up
> recently enough.
>
> <title> tags are structurally simple. They're also required for
> the HTML to be valid, so we have to have one anyway.
>  https://html.spec.whatwg.org/multipage/semantics.html#the-head-element
>
> If we'd like to make a distinction between a "title" and a longer
> "description", then maybe we can adopt a convention similar to
> commit messages: the first line, if it stands alone, is the title,
> and the rest is the description.

I am not interested in making a distinction between a short title and a complete description. I only want the complete description, and worry we won't get it if we call it title.

> I'd rather keep the metadata and its markup as simple as possible.
> It should be easy to write a new test from a blank page once you've
> written 2-3. It should therefore require as little typing and
> memorization as possible; and <meta> tags don't facilitate this at
> all because they're noisy.

I agree with that goal to, and I am not quite sure what to do about the tension between these two positions. What you suggest is probably the way forward, but it still feels a bit wrong.

Titles are mandatory, and we don't care about titles, but we care about something else, so let's just stuff it in the title. I can live with it, but it rubs me the wrong way.

 - Florian


Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

Geoffrey Sneddon-4
On Tue, Apr 12, 2016 at 2:24 AM, Florian Rivoal <[hidden email]> wrote:
>> On Apr 12, 2016, at 02:56, fantasai <[hidden email]> wrote:
>> If we'd like to make a distinction between a "title" and a longer
>> "description", then maybe we can adopt a convention similar to
>> commit messages: the first line, if it stands alone, is the title,
>> and the rest is the description.
>
> I am not interested in making a distinction between a short title and a complete description. I only want the complete description, and worry we won't get it if we call it title.

I'm in agreement with this—I don't think the title matters much. (The
title matters far less often than the filename does when it comes to
finding the file, really.)

>> I'd rather keep the metadata and its markup as simple as possible.
>> It should be easy to write a new test from a blank page once you've
>> written 2-3. It should therefore require as little typing and
>> memorization as possible; and <meta> tags don't facilitate this at
>> all because they're noisy.
>
> I agree with that goal to, and I am not quite sure what to do about the tension between these two positions. What you suggest is probably the way forward, but it still feels a bit wrong.
>
> Titles are mandatory, and we don't care about titles, but we care about something else, so let's just stuff it in the title. I can live with it, but it rubs me the wrong way.

I agree. I don't think we should put it in the title—I still don't see
*why* we care where test authors put the description, though. As far
as I can tell, the only thing anyone cares about is that the
description is there, so I don't see why we should bother to require
it be in any specific form. The policy for w-p-t is essentially that
the reviewer has to be able to understand the test—so a description is
there if it isn't self-evident, somewhere, typically in a comment. A
comment seems easier and more normal syntax for such descriptions than
anything else, and hence less to remember.

/Geoffrey.

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

r12a
On 12/04/2016 18:58, Geoffrey Sneddon wrote:
> there if it isn't self-evident, somewhere, typically in a comment. A
> comment seems easier and more normal syntax for such descriptions than
> anything else, and hence less to remember.

wait, wait....  markup is about labelling data so that it has meaning

for example, it's quite possible that some application will want to pick
up the assertion and do something with it.  I certainly need to be able
to do that for the i18n test suite results.  In that case, it needs to
be labelled as an assertion in some way.

sorry to raise a difficulty.  I'd like a simple solution to this too,
but as i said i agree with Florian that if we use the title element
no-one (well very few people) will give specific enough information in a
title element to allow for proper understanding of the test.

unless the test author has a form-based tool for generating tests, i
think they need to write some markup.  If this were xml we could invent
a simple assertion element in the head next to the title element, but
it's not.  I (unfortunately) can't think of anything shorter and simpler
than a meta element, even if we were to break the semantics.

ri



Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

fantasai
On 04/12/2016 02:15 PM, [hidden email] wrote:

> On 12/04/2016 18:58, Geoffrey Sneddon wrote:
>> there if it isn't self-evident, somewhere, typically in a comment. A
>> comment seems easier and more normal syntax for such descriptions than
>> anything else, and hence less to remember.
>
> wait, wait....  markup is about labelling data so that it has meaning
>
> for example, it's quite possible that some application will want to
> pick up the assertion and do something with it.  I certainly need to
> be able to do that for the i18n test suite results.  In that case,
> it needs to be labelled as an assertion in some way.

Agreed it needs to be machine-readable; therefore better to be in the
markup somehow.

> sorry to raise a difficulty.  I'd like a simple solution to this too,
> but as i said i agree with Florian that if we use the title element
> no-one (well very few people) will give specific enough information
> in a title element to allow for proper understanding of the test.

I think we can solve this by updating the template and having good
examples of tests that write proper assertions in the title. I can
even process all of our existing tests that have an assertion to
move it into the title.

The test template would look like this:

<!DOCTYPE html>
<title>
   Title of your test on one line [optional]

   ... Assertion of your test, i.e. what does passing this test
   prove. E.g. When text-align is not set, its initial value depends
    on dir attribute.) ...
</title>
<link rel="match" href="references/reference-filename.html">
<link rel="help" href="http://www.w3.org/TR/...">
<link rel="help" href="http://www.w3.org/TR/...">
<style>
   ... CSS for test ...
</style>

... body of test ...

It's not exactly a "title" in the <title>, but this solves
several problems:
   * <title> is required for validity, so we have to have one;
     it might as well have a purpose applicable to everybody
   * Description is in an element, which makes it easier to format
     than in an attribute, where long text values are awkward.
   * Less memorization of boilerplate than with <meta> tags.

~fantasai

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

r12a
On 12/04/2016 20:10, fantasai wrote:

> The test template would look like this:
>
> <!DOCTYPE html>
> <title>
>    Title of your test on one line [optional]
>
>    ... Assertion of your test, i.e. what does passing this test
>    prove. E.g. When text-align is not set, its initial value depends
>     on dir attribute.) ...
> </title>

i'm still not keen on this.  It's risky. For example, how do you tell
whether there's a one line short title followed by an assertion
paragraph or no short title but two paragraphs? It also requires dealing
with different line breaks, and there's a risk of lines not being
properly separated, etc.

if we have to use the title element, how about using the title attribute
with it for the optional bit, ie. the short title.  Then at least we can
tell the difference easily.

ie.

<!doctype html>
<html>
<head>
<meta charset="UTF-8">
<title title="short title for your test">Assertion, i.e. in situation X,
Y will happen. E.g. When text-align is not set, its initial value
depends on the dir attribute.</title>


and btw in the documentation i'd say that the title attribute is
optional, but recommended.

ri

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

r12a
In reply to this post by fantasai
On 12/04/2016 20:10, fantasai wrote:
>    * Description is in an element, which makes it easier to format
>      than in an attribute, where long text values are awkward.

i'm closing my eyes to the fact that it's also problematic should we
want a well internationalised solution, given that only plain text is
allowed in the title element.

ri

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

fantasai
In reply to this post by r12a
On 04/12/2016 03:50 PM, [hidden email] wrote:

> On 12/04/2016 20:10, fantasai wrote:
>> The test template would look like this:
>>
>> <!DOCTYPE html>
>> <title>
>>    Title of your test on one line [optional]
>>
>>    ... Assertion of your test, i.e. what does passing this test
>>    prove. E.g. When text-align is not set, its initial value depends
>>     on dir attribute.) ...
>> </title>
>
> i'm still not keen on this.  It's risky. For example, how do you tell whether there's a one line short title followed by an
> assertion paragraph or no short title but two paragraphs? It also requires dealing with different line breaks, and there's a
> risk of lines not being properly separated, etc.
>
> if we have to use the title element, how about using the title attribute with it for the optional bit, ie. the short title.
> Then at least we can tell the difference easily.
>
> ie.
>
> <!doctype html>
> <html>
> <head>
> <meta charset="UTF-8">
> <title title="short title for your test">Assertion, i.e. in situation X, Y will happen. E.g. When text-align is not set, its
> initial value depends on the dir attribute.</title>
>
>
> and btw in the documentation i'd say that the title attribute is optional, but recommended.

This makes sense to me!

~fantasai

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

Florian Rivoal-4

> On Apr 13, 2016, at 06:29, fantasai <[hidden email]> wrote:
>
> On 04/12/2016 03:50 PM, [hidden email] wrote:
>> On 12/04/2016 20:10, fantasai wrote:
>>> The test template would look like this:
>>>
>>> <!DOCTYPE html>
>>> <title>
>>>   Title of your test on one line [optional]
>>>
>>>   ... Assertion of your test, i.e. what does passing this test
>>>   prove. E.g. When text-align is not set, its initial value depends
>>>    on dir attribute.) ...
>>> </title>
>>
>> i'm still not keen on this.  It's risky. For example, how do you tell whether there's a one line short title followed by an
>> assertion paragraph or no short title but two paragraphs? It also requires dealing with different line breaks, and there's a
>> risk of lines not being properly separated, etc.
>>
>> if we have to use the title element, how about using the title attribute with it for the optional bit, ie. the short title.
>> Then at least we can tell the difference easily.
>>
>> ie.
>>
>> <!doctype html>
>> <html>
>> <head>
>> <meta charset="UTF-8">
>> <title title="short title for your test">Assertion, i.e. in situation X, Y will happen. E.g. When text-align is not set, its
>> initial value depends on the dir attribute.</title>
>>
>>
>> and btw in the documentation i'd say that the title attribute is optional, but recommended.
>
> This makes sense to me!

If we're using markup in a way that is unconventional enough that people won't get it without looking at the documentation, I am not sure what we gain over an assert meta.

Also, considering that a key goal is to enable synchronization from browser vendors' test repos, this is not going to be super friendly. Existing tests won't be conforming to this pattern, since it is not standard. So we'd have to fix the tests after importing them, but there's no automated way to detect which tests have an assertion-in-a-title, and which just have an old fashioned title. Also, I don't know for sure, but there is a chance that the title element is already used in some way in the vendors internal systems, and that this usage would be disturbed by us changing it and synchronizing back.

My preference still goes to the assert meta.

* Unlike the flags meta, you don't need to look up the documentation to know how to write it. You need to be made aware that it exists, but once you are, the actual syntax is completely obvious.

* We can programmatically detect which tests don't have one when importing large amount of tests from external sources

* It doesn't go against authoring habits. Sure, people aren't used to writing one, but unlike a title element, they don't have a conflicting preconceived notion of how it should be used.

 - Florian
Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

r12a
On 13/04/2016 01:09, Florian Rivoal wrote:

> If we're using markup in a way that is unconventional enough that people won't get it without looking at the documentation, I am not sure what we gain over an assert meta.
>
> Also, considering that a key goal is to enable synchronization from browser vendors' test repos, this is not going to be super friendly. Existing tests won't be conforming to this pattern, since it is not standard. So we'd have to fix the tests after importing them, but there's no automated way to detect which tests have an assertion-in-a-title, and which just have an old fashioned title. Also, I don't know for sure, but there is a chance that the title element is already used in some way in the vendors internal systems, and that this usage would be disturbed by us changing it and synchronizing back.
>
> My preference still goes to the assert meta.
>
> * Unlike the flags meta, you don't need to look up the documentation to know how to write it. You need to be made aware that it exists, but once you are, the actual syntax is completely obvious.
>
> * We can programmatically detect which tests don't have one when importing large amount of tests from external sources
>
> * It doesn't go against authoring habits. Sure, people aren't used to writing one, but unlike a title element, they don't have a conflicting preconceived notion of how it should be used.

+1 to all that. Thanks Florian.

Just so that it's clear, my suggestion about title on title was only an
attempt to prevent us ending up even further out there in the direction
of what i still think is a bad solution – actually i agree with Florian
that repurposing the title element content to save just a small number
of keystrokes is not a good idea, and in addition is not worth the
refactoring work that it will involve.

ri

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

r12a
In reply to this post by fantasai
On 12/04/2016 20:10, fantasai wrote:
> <link rel="help" href="http://www.w3.org/TR/...">
> <link rel="help" href="http://www.w3.org/TR/...">

actually, having to add this specific help metadata is always one of the
things that most slows me down and irks me when creating tests. It's
certainly important information, but can't we make the link
automatically by using a directory structure that matches the document
structure.  That's something that would make a real difference to the
ease of writing tests.

ie. all tests for the section "7.1. Text Alignment: the text-align
shorthand" get added to a subdirectory at

css-text-3/justification/text-align-property/

the tools can then use the directory structure to identify the section
to which the test belongs, and bingo you've reduced the work of writing
tests to a far greater degree than worrying about one meta element.

it also has advantages if the document structure changes. Just move the
tests to a different directory - no need to change the metadata or
update the file contents everywhere.

it also makes it easier for people to locate tests for a particular
feature by looking in the repo, rather than having to find other
locations that provide such info (if there are any) and tracking down
tests from there.

it also removes the question of whether my test should point to the TR
version of the document or the ED, and btw ask yourself what's the
current ED location today.

maybe i'm missing something, but handling this automatically seems like
a real effort-saver for people creating tests.

ri

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

fantasai
On 04/13/2016 07:50 AM, [hidden email] wrote:

> On 12/04/2016 20:10, fantasai wrote:
>> <link rel="help" href="http://www.w3.org/TR/...">
>> <link rel="help" href="http://www.w3.org/TR/...">
>
> actually, having to add this specific help metadata is always one of the things that most slows me down and irks me when
> creating tests. It's certainly important information, but can't we make the link automatically by using a directory structure
> that matches the document structure.  That's something that would make a real difference to the ease of writing tests.
>
> ie. all tests for the section "7.1. Text Alignment: the text-align shorthand" get added to a subdirectory at
>
> css-text-3/justification/text-align-property/
>
> the tools can then use the directory structure to identify the section to which the test belongs, and bingo you've reduced the
> work of writing tests to a far greater degree than worrying about one meta element.
>
> it also has advantages if the document structure changes. Just move the tests to a different directory - no need to change the
> metadata or update the file contents everywhere.
>
> it also makes it easier for people to locate tests for a particular feature by looking in the repo, rather than having to find
> other locations that provide such info (if there are any) and tracking down tests from there.
>
> it also removes the question of whether my test should point to the TR version of the document or the ED, and btw ask yourself
> what's the current ED location today.
>
> maybe i'm missing something, but handling this automatically seems like a real effort-saver for people creating tests.

Well, there are two problems with this
   * We have tests that test the interaction among sections.
     These are quite important; they also don't fit into a
     hierarchical structure.
   * This requires everyone who uses our tests to mirror
     our exact file structure.
   * It centralizes control over the file structure to our
     mapping file, making that a blocker for adding new
     directories in the test suite.

Wrt document structure changes -- the CSSWG is in the habit
of providing explicit ID markers to all headings in our spec
sources, and keeping those stable even as the spec structure
changes. So that's not really an issue here.

Now, we have had some concerns about too many test files in
one directory, and making it easier to find tests. So I'm
perfectly happy if we want to set up subdirectories to
organize the tests according to their "primary" section.

I'm also alright if we want to associate a default link to
that section with, e.g. a LINK file containing the URL in
that directory. The build system can auto-inject the help
link when building the test suite, and everyone else can
look at the LINK file for the directory. (That said, I'm
a little concerned however that the appropriate granularity
for directory structure will be coarser than subsections.)

Not really OK with creating a canonical directory structure
and then relying on that as the sole information for
associating tests and spec sections, for the reasons above.

Wrt TR vs ED -- Links should be to the TR version, and we can
fix that up once in awhile if people forget. (Also, going
forward, Echidna will allow us to keep TR much more up-to-date,
so there will (hopefully) be no reason to continue to use EDs
for day-to-date use.)

~fantasai

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

Geoffrey Sneddon-4
On Wed, Apr 13, 2016 at 6:23 PM, fantasai <[hidden email]> wrote:

> On 04/13/2016 07:50 AM, [hidden email] wrote:
>>
>> On 12/04/2016 20:10, fantasai wrote:
>>>
>>> <link rel="help" href="http://www.w3.org/TR/...">
>>> <link rel="help" href="http://www.w3.org/TR/...">
>>
>>
>> actually, having to add this specific help metadata is always one of the
>> things that most slows me down and irks me when
>> creating tests. It's certainly important information, but can't we make
>> the link automatically by using a directory structure
>> that matches the document structure.  That's something that would make a
>> real difference to the ease of writing tests.
>>
>> ie. all tests for the section "7.1. Text Alignment: the text-align
>> shorthand" get added to a subdirectory at
>>
>> css-text-3/justification/text-align-property/
>>
>> the tools can then use the directory structure to identify the section to
>> which the test belongs, and bingo you've reduced the
>> work of writing tests to a far greater degree than worrying about one meta
>> element.
>>
>> it also has advantages if the document structure changes. Just move the
>> tests to a different directory - no need to change the
>> metadata or update the file contents everywhere.
>>
>> it also makes it easier for people to locate tests for a particular
>> feature by looking in the repo, rather than having to find
>> other locations that provide such info (if there are any) and tracking
>> down tests from there.
>>
>> it also removes the question of whether my test should point to the TR
>> version of the document or the ED, and btw ask yourself
>> what's the current ED location today.
>>
>> maybe i'm missing something, but handling this automatically seems like a
>> real effort-saver for people creating tests.
>
>
> Well, there are two problems with this
>   * We have tests that test the interaction among sections.
>     These are quite important; they also don't fit into a
>     hierarchical structure.
>   * This requires everyone who uses our tests to mirror
>     our exact file structure.
>   * It centralizes control over the file structure to our
>     mapping file, making that a blocker for adding new
>     directories in the test suite.

I think you can't count to three. :)

I think everyone is happy to mirror such a directory structure (it is,
after all, roughly what everyone does today). I don't think it
requires people who use the tests to mirror the structure more than
the current situation does—if you want two-way mirroring, you
essentially have to use what the upstream repo uses anyway.

I don't see how it makes us reliant on a mapping file, though?

/Geoffrey

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

fantasai
On 04/13/2016 01:49 PM, Geoffrey Sneddon wrote:
>
> I think you can't count to three. :)

Caching is a Hard Problem. ;)

> I think everyone is happy to mirror such a directory structure (it is,
> after all, roughly what everyone does today). I don't think it
> requires people who use the tests to mirror the structure more than
> the current situation does—if you want two-way mirroring, you
> essentially have to use what the upstream repo uses anyway.

Sure, but we don't have two-way mirroring yet. We do have one-way
mirroring atm, though, so it'd be nice if that continued to work.

> I don't see how it makes us reliant on a mapping file, though?

If we decide to omit the primary <link>, we'd need a mapping
from directory to spec section.

I suppose we could automate one from the fragment IDs, though.

~fantasai

Reply | Threaded
Open this post in threaded view
|

Re: Towards a better testsuite: Metadata

Gérard Talbot-2
In reply to this post by Florian Rivoal-4
Le 2016-04-10 20:56, Florian Rivoal a écrit :

>> On Apr 9, 2016, at 02:00, fantasai <[hidden email]>
>> wrote:
>>
>> I think we can simplify things down to:
>>
>> <!DOCTYPE html>
>> <title>Explain *exactly* what situation you're testing
>> and what it's supposed to do</title>
>> <link rel="help" href="spec">
>> <link rel="match" href="reference">
>>
>> plus optional <meta name=flags> if needed for that test.
>> (I agree with trimming down the flags as well; many are
>> not really necessary at this point.)
>
> Works for me. The only point I am not 100% sure about is putting the
> assertion/explanation in the title, as I think people have
> expectations that a title should be a few words, rather than one or
> two full sentences, and I worry a bit that this set up will give us
> under-described tests.
>
> I think I'd prefer keeping the assertion in an assert meta, and
> effectively disregarding what goes into the title, but I don't know
> what's easier to teach and enforce:
> - Writing long descriptive titles
> - Having an assert meta
>
> But either way, this is the right amount of information. We can
> discuss a little bit about whether this is the ideal way to mark it up
> or not, but if that's that format that people will accept I'm fine
> with it.
>
>  - Florian

Regarding the <title> text.

Here's what I wrote 4 years ago:

{
The title text should not replace the assert or comments in the code.

James Hopkins had a good system. He mentioned the property name, then an
hyphen and then a few other words like other property names.

E.g.
<title>CSS Test: overflow - max-width and percentage</title>

<title>CSS Test: list-style-position - text-indent</title>
}

http://lists.w3.org/Archives/Public/public-css-testsuite/2012Apr/0033.html

With that way of writing the <title> text, the title is generally short,
descriptive, clear, useful and helps searching.

<title>CSS [Specification name] Test: [main property name] - [secondary
property name]</title>

I have used consistently such way of writing the <title> text in the
last 4 years too; that way, I did not have to think much. The <title>
text only lists the main "ingredients" of the test.

We used such system when doing the writing modes tests: go to

http://test.csswg.org/source/css-writing-modes-3/

and then look at the right-most column for the title texts.

Gérard