Submit an Implementation report based on a nightly-unstable test suite: possible?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Submit an Implementation report based on a nightly-unstable test suite: possible?

Gérard Talbot-3
Peter,

I've read

http://lists.w3.org/Archives/Public/public-css-testsuite/2010Aug/0020.html

https://wiki.csswg.org/test/implementation-report

I would be ready and interested to enter test results of Prince 10 rev 3
for the CSS3 Writing Modes test suite *nightly-unstable* build and
submit an implementation report.

1- Is this possible?

2- If so, then what would be the DATESTAMP:
# http://test.csswg.org/suites/css-writing-modes-3_dev/DATESTAMP/

3-
When a test line has many characters in its 2nd (middle) column (in the
column between testname and result), what does it mean?
eg

html/abs-pos-non-replaced-icb-vlr-003.htm d3088b2113ff9dc88028dc89b4d030a83db59a2b ?
xhtml1/abs-pos-non-replaced-icb-vlr-003.xht d3088b2113ff9dc88028dc89b4d030a83db59a2b ?
html/abs-pos-non-replaced-icb-vlr-005.htm b959f5e9a4d09adc6784d74741b43ff0ac1ea0ca ?
xhtml1/abs-pos-non-replaced-icb-vlr-005.xht b959f5e9a4d09adc6784d74741b43ff0ac1ea0ca ?

This comes from the implementation report template for the
nightly-unstable test suite:

http://test.csswg.org/suites/css-writing-modes-3_dev/nightly-unstable/implementation-report-TEMPLATE.data

Gérard
--
Test Format Guidelines
http://testthewebforward.org/docs/test-format-guidelines.html

Test Style Guidelines
http://testthewebforward.org/docs/test-style-guidelines.html

Test Templates
http://testthewebforward.org/docs/test-templates.html

CSS Naming Guidelines
http://testthewebforward.org/docs/css-naming.html

Test Review Checklist
http://testthewebforward.org/docs/review-checklist.html

CSS Metadata
http://testthewebforward.org/docs/css-metadata.html

Reply | Threaded
Open this post in threaded view
|

Re: Submit an Implementation report based on a nightly-unstable test suite: possible?

Florian Rivoal-4

> On 15 Aug 2015, at 05:35, Gérard Talbot <[hidden email]> wrote:
>
> Peter,
>
> I've read
>
> http://lists.w3.org/Archives/Public/public-css-testsuite/2010Aug/0020.html
>
> https://wiki.csswg.org/test/implementation-report
>
> I would be ready and interested to enter test results of Prince 10 rev 3 for the CSS3 Writing Modes test suite *nightly-unstable* build and submit an implementation report.
>
> 1- Is this possible?

I don't know what the official rule is, but I think as long as the build is available so that it is possible for others to reproduce the results should they want to, I believe we should accept it.

 - Florian
Reply | Threaded
Open this post in threaded view
|

Re: Submit an Implementation report based on a nightly-unstable test suite: possible?

Gérard Talbot-3
Le 2015-08-15 06:15, Florian Rivoal a écrit :

>> On 15 Aug 2015, at 05:35, Gérard Talbot <[hidden email]>
>> wrote:
>>
>> Peter,
>>
>> I've read
>>
>> http://lists.w3.org/Archives/Public/public-css-testsuite/2010Aug/0020.html
>>
>> https://wiki.csswg.org/test/implementation-report
>>
>> I would be ready and interested to enter test results of Prince 10 rev
>> 3 for the CSS3 Writing Modes test suite *nightly-unstable* build and
>> submit an implementation report.
>>
>> 1- Is this possible?
>
> I don't know what the official rule is, but I think as long as the
> build is available so that it is possible for others to reproduce the
> results should they want to, I believe we should accept it.
>
>  - Florian

The tests in the nightly-unstable test suite are regularly modified from
day to day, new tests are added every day, some other tests are removed.
So, I now think it would not be possible.

But the entrance page of the nightly-unstable test suite has a link to
implementation report.

http://test.csswg.org/suites/css-writing-modes-3_dev/nightly-unstable/#implement

It's probably an oversight.

Gérard
--
Test Format Guidelines
http://testthewebforward.org/docs/test-format-guidelines.html

Test Style Guidelines
http://testthewebforward.org/docs/test-style-guidelines.html

Test Templates
http://testthewebforward.org/docs/test-templates.html

CSS Naming Guidelines
http://testthewebforward.org/docs/css-naming.html

Test Review Checklist
http://testthewebforward.org/docs/review-checklist.html

CSS Metadata
http://testthewebforward.org/docs/css-metadata.html

Reply | Threaded
Open this post in threaded view
|

Re: Submit an Implementation report based on a nightly-unstable test suite: possible?

Peter Linss
In reply to this post by Gérard Talbot-3
On Aug 14, 2015, at 8:35 PM, Gérard Talbot <[hidden email]> wrote:

Peter,

I've read

http://lists.w3.org/Archives/Public/public-css-testsuite/2010Aug/0020.html

https://wiki.csswg.org/test/implementation-report

I would be ready and interested to enter test results of Prince 10 rev 3 for the CSS3 Writing Modes test suite *nightly-unstable* build and submit an implementation report.

1- Is this possible?

Yes.


2- If so, then what would be the DATESTAMP:
# http://test.csswg.org/suites/css-writing-modes-3_dev/DATESTAMP/

That field isn’t really needed anymore. It was intended to be the date/time (in ISO yyyy-mm-dd hh:mm:ss format) of the time the results were recorded.


3-
When a test line has many characters in its 2nd (middle) column (in the column between testname and result), what does it mean?
eg

html/abs-pos-non-replaced-icb-vlr-003.htm d3088b2113ff9dc88028dc89b4d030a83db59a2b ?
xhtml1/abs-pos-non-replaced-icb-vlr-003.xht d3088b2113ff9dc88028dc89b4d030a83db59a2b ?
html/abs-pos-non-replaced-icb-vlr-005.htm b959f5e9a4d09adc6784d74741b43ff0ac1ea0ca ?
xhtml1/abs-pos-non-replaced-icb-vlr-005.xht b959f5e9a4d09adc6784d74741b43ff0ac1ea0ca ?

The second column is the changeset ID that the test was last modified in. Just leave it in place, it let’s the test harness associate the result with the exact revision of the test.


There are two ways that you can submit tests for Prince, you can either generate an implementation report using the template that you’ve found, or you can set the proper user agent string in the harness and just run the tests.

People with appropriate permissions (you are one of them) can upload the implementation reports yourself directly into the test harness. On the “Run Tests” page first click the “Change” link at the top pf the page to change the user agent to the one you’re entering test for (if it’s not the browser you’re using), then click the “Batch Upload” button on the “Run Tests” page. Enter the date/time and upload the file.

When you set the user agent, you can either pick the user agent you want from the list (if it’s present), or you can enter the entire user agent string of the agent. For Prince it should be something like:
Prince/10.0 (http://www.princexml.com; Linux i686)

Alternatively you can just email the implementation report to me and I’ll upload it for you.

Peter

signature.asc (506 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Submit an Implementation report based on a nightly-unstable test suite: possible?

Peter Linss
In reply to this post by Gérard Talbot-3

> On Aug 15, 2015, at 11:26 AM, Gérard Talbot <[hidden email]> wrote:
>
> Le 2015-08-15 06:15, Florian Rivoal a écrit :
>>> On 15 Aug 2015, at 05:35, Gérard Talbot <[hidden email]> wrote:
>>> Peter,
>>> I've read
>>> http://lists.w3.org/Archives/Public/public-css-testsuite/2010Aug/0020.html
>>> https://wiki.csswg.org/test/implementation-report
>>> I would be ready and interested to enter test results of Prince 10 rev 3 for the CSS3 Writing Modes test suite *nightly-unstable* build and submit an implementation report.
>>> 1- Is this possible?
>> I don't know what the official rule is, but I think as long as the
>> build is available so that it is possible for others to reproduce the
>> results should they want to, I believe we should accept it.
>> - Florian
>
> The tests in the nightly-unstable test suite are regularly modified from day to day, new tests are added every day, some other tests are removed. So, I now think it would not be possible.
>
> But the entrance page of the nightly-unstable test suite has a link to implementation report.
>
> http://test.csswg.org/suites/css-writing-modes-3_dev/nightly-unstable/#implement
>
> It's probably an oversight.

I believe Florian was referring to the build of Prince, not the build of the test suite.

His point was that we accept results from any CSS implementation that’s available to the public so that the results can be reproduced.

Note that the test harness associates each entered result with the version of the individual test. So while the test suite is rebuilt each night, individual tests don’t tend to change that often. Any result entered will remain as long as the test doesn’t change.

Peter

signature.asc (506 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Submit an Implementation report based on a nightly-unstable test suite: possible?

Gérard Talbot-3
In reply to this post by Peter Linss
Le 2015-08-15 21:06, Linss, Peter a écrit :

> On Aug 14, 2015, at 8:35 PM, Gérard Talbot <[hidden email]>
> wrote:
>>
>> Peter,
>>
>> I've read
>>
>> http://lists.w3.org/Archives/Public/public-css-testsuite/2010Aug/0020.html
>>
>> https://wiki.csswg.org/test/implementation-report
>>
>> I would be ready and interested to enter test results of Prince 10 rev
>> 3 for the CSS3 Writing Modes test suite *nightly-unstable* build and
>> submit an implementation report.
>>
>> 1- Is this possible?
>
> Yes.
>
>>
>> 2- If so, then what would be the DATESTAMP:
>> # http://test.csswg.org/suites/css-writing-modes-3_dev/DATESTAMP/ 
>> <http://test.csswg.org/suites/css-writing-modes-3_dev/DATESTAMP/>
>
> That field isn’t really needed anymore. It was intended to be the
> date/time (in ISO yyyy-mm-dd hh:mm:ss format) of the time the results
> were recorded.
>

Isn't that DATESTAMP field associated with the alpha or beta or CR
status of the test suite?

# http://test.csswg.org/suites/css2.1/20100917/
is RC1

while

# http://test.csswg.org/suites/css2.1/20110323/
is RC6

If I submit an implementation report for Prince 10 rev 4, then should I
remove that DATESTAMP line entirely or just leave it as

# http://test.csswg.org/suites/css-writing-modes-3_dev/DATESTAMP/ 
<http://test.csswg.org/suites/css-writing-modes-3_dev/DATESTAMP/>

?

>>
>> 3-
>> When a test line has many characters in its 2nd (middle) column (in
>> the column between testname and result), what does it mean?
>> eg
>>
>> html/abs-pos-non-replaced-icb-vlr-003.htm d3088b2113ff9dc88028dc89b4d030a83db59a2b ?
>> xhtml1/abs-pos-non-replaced-icb-vlr-003.xht d3088b2113ff9dc88028dc89b4d030a83db59a2b ?
>> html/abs-pos-non-replaced-icb-vlr-005.htm b959f5e9a4d09adc6784d74741b43ff0ac1ea0ca ?
>> xhtml1/abs-pos-non-replaced-icb-vlr-005.xht b959f5e9a4d09adc6784d74741b43ff0ac1ea0ca ?
>
> The second column is the changeset ID that the test was last modified
> in. Just leave it in place, it let’s the test harness associate the
> result with the exact revision of the test.
>

Okay... it makes sense.

>
> There are two ways that you can submit tests for Prince, you can
> either generate an implementation report using the template that
> you’ve found, or you can set the proper user agent string in the
> harness and just run the tests.
>
> People with appropriate permissions (you are one of them) can upload
> the implementation reports yourself directly into the test harness. On
> the “Run Tests” page first click the “Change” link at the top pf the
> page to change the user agent to the one you’re entering test for (if
> it’s not the browser you’re using), then click the “Batch Upload”
> button on the “Run Tests” page. Enter the date/time and upload the
> file.
>
> When you set the user agent, you can either pick the user agent you
> want from the list (if it’s present), or you can enter the entire user
> agent string of the agent. For Prince it should be something like:
> Prince/10.0 (http://www.princexml.com; Linux i686)

The 'Linux i686' string is not fetchable... otherwise I do not know how
you got it.

I have tried to query all possible navigator attributes for Prince and
never got the complete infor what you indicate here:

Prince can not handle the ListAllNavigatorAttributesAndMethods()
function I have in

http://www.gtalbot.org/DHTMLSection/ListAllAttributesAndMethodsOfObjects.html

So, I created

http://www.gtalbot.org/BrowserBugsSection/CSS3WritingModes/query-navigator-attributes.html

and I use in command line

~$ prince --javascript query-navigator-attributes.html

and I still do not get os and cpu attributes or build number which would
be useful.

I guess

~$ prince --version

info is sufficient.


> Alternatively you can just email the implementation report to me and
> I’ll upload it for you.
>
> Peter

Peter, thank you for your complete response. I appreciate this.

Prince 10 rev 4 does not support 'text-orientation', it does not support
'text-combine-upright' and it only support 'writing-mode: vertical-rl' .
So, the number of tests with useful+revealing test results or worth
testing for Prince is about 150 tests.

Right now, the first 6 lines of the implementation report I intend to
submit look like this:

# Prince 10 rev 4 Kubuntu Linux 14.04.3 LTS
# Prince/10 (www.princexml.com)
# http://test.csswg.org/suites/css-writing-modes-3_dev/DATESTAMP/ 
<http://test.csswg.org/suites/css-writing-modes-3_dev/DATESTAMP/>
testname revision result comment
html/abs-pos-non-replaced-icb-vlr-003.htm d3088b2113ff9dc88028dc89b4d030a83db59a2b na (vertical-lr
not implemented)
xhtml1/abs-pos-non-replaced-icb-vlr-003.xht d3088b2113ff9dc88028dc89b4d030a83db59a2b na (vertical-lr
not implemented)
...

Gérard
--
Test Format Guidelines
http://testthewebforward.org/docs/test-format-guidelines.html

Test Style Guidelines
http://testthewebforward.org/docs/test-style-guidelines.html

Test Templates
http://testthewebforward.org/docs/test-templates.html

CSS Naming Guidelines
http://testthewebforward.org/docs/css-naming.html

Test Review Checklist
http://testthewebforward.org/docs/review-checklist.html

CSS Metadata
http://testthewebforward.org/docs/css-metadata.html

Reply | Threaded
Open this post in threaded view
|

Re: Submit an Implementation report based on a nightly-unstable test suite: possible?

Gérard Talbot-3
In reply to this post by Peter Linss
Le 2015-08-15 21:09, Linss, Peter a écrit :

>> On Aug 15, 2015, at 11:26 AM, Gérard Talbot
>> <[hidden email]> wrote:
>>
>> Le 2015-08-15 06:15, Florian Rivoal a écrit :
>>>> On 15 Aug 2015, at 05:35, Gérard Talbot <[hidden email]>
>>>> wrote:
>>>> Peter,
>>>> I've read
>>>> http://lists.w3.org/Archives/Public/public-css-testsuite/2010Aug/0020.html
>>>> https://wiki.csswg.org/test/implementation-report
>>>> I would be ready and interested to enter test results of Prince 10
>>>> rev 3 for the CSS3 Writing Modes test suite *nightly-unstable* build
>>>> and submit an implementation report.
>>>> 1- Is this possible?
>>> I don't know what the official rule is, but I think as long as the
>>> build is available so that it is possible for others to reproduce the
>>> results should they want to, I believe we should accept it.
>>> - Florian
>>
>> The tests in the nightly-unstable test suite are regularly modified
>> from day to day, new tests are added every day, some other tests are
>> removed. So, I now think it would not be possible.
>>
>> But the entrance page of the nightly-unstable test suite has a link to
>> implementation report.
>>
>> http://test.csswg.org/suites/css-writing-modes-3_dev/nightly-unstable/#implement
>>
>> It's probably an oversight.
>
>
> I believe Florian was referring to the build of Prince, not the build
> of the test suite.

Okay.

There is no way to fetch the build number of Prince. The only version
number that I believe that can be queried is

~$ prince --version


Gérard

> His point was that we accept results from any CSS implementation
> that’s available to the public so that the results can be reproduced.
>
> Note that the test harness associates each entered result with the
> version of the individual test. So while the test suite is rebuilt
> each night, individual tests don’t tend to change that often. Any
> result entered will remain as long as the test doesn’t change.
>
> Peter

--
Test Format Guidelines
http://testthewebforward.org/docs/test-format-guidelines.html

Test Style Guidelines
http://testthewebforward.org/docs/test-style-guidelines.html

Test Templates
http://testthewebforward.org/docs/test-templates.html

CSS Naming Guidelines
http://testthewebforward.org/docs/css-naming.html

Test Review Checklist
http://testthewebforward.org/docs/review-checklist.html

CSS Metadata
http://testthewebforward.org/docs/css-metadata.html

Reply | Threaded
Open this post in threaded view
|

Re: Submit an Implementation report based on a nightly-unstable test suite: possible?

Peter Linss
In reply to this post by Gérard Talbot-3

On Aug 16, 2015, at 2:16 PM, Gérard Talbot <[hidden email]> wrote:

Le 2015-08-15 21:06, Linss, Peter a écrit :
On Aug 14, 2015, at 8:35 PM, Gérard Talbot <[hidden email]> wrote:
Peter,
I've read
http://lists.w3.org/Archives/Public/public-css-testsuite/2010Aug/0020.html
https://wiki.csswg.org/test/implementation-report
I would be ready and interested to enter test results of Prince 10 rev 3 for the CSS3 Writing Modes test suite *nightly-unstable* build and submit an implementation report.
1- Is this possible?
Yes.
2- If so, then what would be the DATESTAMP:
# http://test.csswg.org/suites/css-writing-modes-3_dev/DATESTAMP/ <http://test.csswg.org/suites/css-writing-modes-3_dev/DATESTAMP/>
That field isn’t really needed anymore. It was intended to be the
date/time (in ISO yyyy-mm-dd hh:mm:ss format) of the time the results
were recorded.

Isn't that DATESTAMP field associated with the alpha or beta or CR status of the test suite?

# http://test.csswg.org/suites/css2.1/20100917/
is RC1

while

# http://test.csswg.org/suites/css2.1/20110323/
is RC6

Sorry, you're correct, that datestamp was for the release of the test suite (this was only used back when we were trying to ship 2.1)

If I submit an implementation report for Prince 10 rev 4, then should I remove that DATESTAMP line entirely or just leave it as

# http://test.csswg.org/suites/css-writing-modes-3_dev/DATESTAMP/ <http://test.csswg.org/suites/css-writing-modes-3_dev/DATESTAMP/>

?

The line is just seen as a comment and ignored by the current import script, so it doesn't matter at all.


3-
When a test line has many characters in its 2nd (middle) column (in the column between testname and result), what does it mean?
eg
html/abs-pos-non-replaced-icb-vlr-003.htm d3088b2113ff9dc88028dc89b4d030a83db59a2b ?
xhtml1/abs-pos-non-replaced-icb-vlr-003.xht d3088b2113ff9dc88028dc89b4d030a83db59a2b ?
html/abs-pos-non-replaced-icb-vlr-005.htm b959f5e9a4d09adc6784d74741b43ff0ac1ea0ca ?
xhtml1/abs-pos-non-replaced-icb-vlr-005.xht b959f5e9a4d09adc6784d74741b43ff0ac1ea0ca ?
The second column is the changeset ID that the test was last modified
in. Just leave it in place, it let’s the test harness associate the
result with the exact revision of the test.

Okay... it makes sense.

There are two ways that you can submit tests for Prince, you can
either generate an implementation report using the template that
you’ve found, or you can set the proper user agent string in the
harness and just run the tests.
People with appropriate permissions (you are one of them) can upload
the implementation reports yourself directly into the test harness. On
the “Run Tests” page first click the “Change” link at the top pf the
page to change the user agent to the one you’re entering test for (if
it’s not the browser you’re using), then click the “Batch Upload”
button on the “Run Tests” page. Enter the date/time and upload the
file.
When you set the user agent, you can either pick the user agent you
want from the list (if it’s present), or you can enter the entire user
agent string of the agent. For Prince it should be something like:
Prince/10.0 (http://www.princexml.com; Linux i686)

The 'Linux i686' string is not fetchable... otherwise I do not know how you got it.

I got that string by looking up old ones in the test harness and just replacing the version number. I'm not sure where they originally came from, I probably just synthesized them so that the harness would understand them.


I have tried to query all possible navigator attributes for Prince and never got the complete infor what you indicate here:

Prince can not handle the ListAllNavigatorAttributesAndMethods() function I have in

http://www.gtalbot.org/DHTMLSection/ListAllAttributesAndMethodsOfObjects.html

So, I created

http://www.gtalbot.org/BrowserBugsSection/CSS3WritingModes/query-navigator-attributes.html

and I use in command line

~$ prince --javascript query-navigator-attributes.html

and I still do not get os and cpu attributes or build number which would be useful.

I guess

~$ prince --version

info is sufficient.

Yeah, just give the best info you can find. If you make it look like a "normal" UA string then the harness can parse it to get the version and OS info. So just enter something that looks like what I listed but put in the best version info you have.



Alternatively you can just email the implementation report to me and
I’ll upload it for you.
Peter

Peter, thank you for your complete response. I appreciate this.

Prince 10 rev 4 does not support 'text-orientation', it does not support 'text-combine-upright' and it only support 'writing-mode: vertical-rl' . So, the number of tests with useful+revealing test results or worth testing for Prince is about 150 tests.

Right now, the first 6 lines of the implementation report I intend to submit look like this:

# Prince 10 rev 4 Kubuntu Linux 14.04.3 LTS
# Prince/10 (www.princexml.com)
# http://test.csswg.org/suites/css-writing-modes-3_dev/DATESTAMP/ <http://test.csswg.org/suites/css-writing-modes-3_dev/DATESTAMP/>
testname revision result comment
html/abs-pos-non-replaced-icb-vlr-003.htm d3088b2113ff9dc88028dc89b4d030a83db59a2b na (vertical-lr not implemented)
xhtml1/abs-pos-non-replaced-icb-vlr-003.xht d3088b2113ff9dc88028dc89b4d030a83db59a2b na (vertical-lr not implemented)
...

You don't have to have entries for every test in the report, just whatever you have results for. Any line with a "na" result will just be ignored anyway.
Peter

signature.asc (506 bytes) Download Attachment