libwww contributions

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

libwww contributions

Vic Bancroft-2

We are approaching a release of libwww version 5.4.1, since there are quite a few accumulated updates to the source tree which should be published. / With the following ChangeLog entry, we should be ready to push out what we have now . . .
/

    /[bancroft@hilbert libwww]$ cvs diff ChangeLog/
    /Index: ChangeLog/
    /===================================================================/
    /RCS file: /sources/public/libwww/ChangeLog,v/
    /retrieving revision 1.53/
    /diff -r1.53 ChangeLog/
    /2a3,9/
    /> 2005-11-11 Vic Bancroft <[hidden email]>/
    />/
    /> * Library/src/HT*.html, Library/src/SSL/HT*.html: wrap/
    /> all header files with extern "C"/
    /> * Library/src/HTFile, configure.ac: add a basis for/
    /> addressing Ben's security concerns/
    />/
    //

//There are at least two outstanding sets of modifications, for which we have not received patches.  The first is usage of the new function in HTFile to address the potential buffer overflow as discussed by Ben.  The second is the set of changes as volunteered by Alec.
/
/On February 02, 2004 12:41 PM, Alec wrote,

> We have made some modifications to the code that we would be happy to
> contribute to the tree if others agree.  They include:
>
> 1) Parallel fetch from a single host (the solutions discussed on the
> list did not work without further modification when the internal cache
> was used)
> 2) The HTML parser was not configured to parse <SCRIPT> blocks correctly
> (in HTMLPDTD.c SCRIPT was defined as SGML_MIXED, changing it to
> SGML_LITERAL fixed the issues)
> 3) There were some issues related to cookies that I had to fix, related
> to cookies being sent to the wrong hosts when a redirect to a URL in a
> different domain was encountered.  The net result was that multiple
> cookie headers were sent, instead of just the one.

If we reduce the lag time between releases, perhaps will be easier to incorporate modifications to the source tree into package releases to the various applications and distributions.  

Unless there are some objections, I would like to invite Jose to schedule some time to help make the 5.4.1 release!

more,
l8r,
v


--
"The future is here. It's just not evenly distributed yet."
 -- William Gibson, quoted by Whitfield Diffie


Reply | Threaded
Open this post in threaded view
|

Re: libwww contributions

Arthur Smith

I had trouble compiling the latest (CVS) under solaris, anybody else
tried this? Do you have a set of test OS's to validate before release (I
can test on Linux, BSD, MacOS X, Solaris at least).

          Arthur

Vic Bancroft wrote:

>
> We are approaching a release of libwww version 5.4.1, since there are
> quite a few accumulated updates to the source tree which should be
> published. / With the following ChangeLog entry, we should be ready to
> push out what we have now . . . /
>
>    /[bancroft@hilbert libwww]$ cvs diff ChangeLog/
>    /Index: ChangeLog/
>    /===================================================================/
>    /RCS file: /sources/public/libwww/ChangeLog,v/
>    /retrieving revision 1.53/
>    /diff -r1.53 ChangeLog/
>    /2a3,9/
>    /> 2005-11-11 Vic Bancroft <[hidden email]>/
>    />/
>    /> * Library/src/HT*.html, Library/src/SSL/HT*.html: wrap/
>    /> all header files with extern "C"/
>    /> * Library/src/HTFile, configure.ac: add a basis for/
>    /> addressing Ben's security concerns/
>    />/
>    //
>
> //There are at least two outstanding sets of modifications, for which
> we have not received patches.  The first is usage of the new function
> in HTFile to address the potential buffer overflow as discussed by
> Ben.  The second is the set of changes as volunteered by Alec.
> /
> /On February 02, 2004 12:41 PM, Alec wrote,
>
>> We have made some modifications to the code that we would be happy to
>> contribute to the tree if others agree.  They include:
>>
>> 1) Parallel fetch from a single host (the solutions discussed on the
>> list did not work without further modification when the internal cache
>> was used)
>> 2) The HTML parser was not configured to parse <SCRIPT> blocks correctly
>> (in HTMLPDTD.c SCRIPT was defined as SGML_MIXED, changing it to
>> SGML_LITERAL fixed the issues)
>> 3) There were some issues related to cookies that I had to fix, related
>> to cookies being sent to the wrong hosts when a redirect to a URL in a
>> different domain was encountered.  The net result was that multiple
>> cookie headers were sent, instead of just the one.
>
>
> If we reduce the lag time between releases, perhaps will be easier to
> incorporate modifications to the source tree into package releases to
> the various applications and distributions.
> Unless there are some objections, I would like to invite Jose to
> schedule some time to help make the 5.4.1 release!
>
> more,
> l8r,
> v
>
>


Reply | Threaded
Open this post in threaded view
|

Re: libwww contributions

Vic Bancroft-2

Arthur Smith wrote:

> I had trouble compiling the latest (CVS) under solaris, anybody else
> tried this?

Ouch, could you send a compile log ?

I have been using the library primarily under some hybrid x86_64 linux . . .

> Do you have a set of test OS's to validate before release (I can test
> on Linux, BSD, MacOS X, Solaris at least).

Wow, that would really help !

more,
l8r,
v

--
"The future is here. It's just not evenly distributed yet."
 -- William Gibson, quoted by Whitfield Diffie


Reply | Threaded
Open this post in threaded view
|

Re: libwww contributions

Vic Bancroft-2
In reply to this post by Vic Bancroft-2

Alec H. Peterson wrote:

> On Nov 28, 2005, at 7:01, Vic Bancroft wrote:
>
>> There are at least two outstanding sets of modifications, for  which
>> we have not received patches.  The first is usage of the new  
>> function in HTFile to address the potential buffer overflow as  
>> discussed by Ben.  The second is the set of changes as volunteered  
>> by Alec.
>
> Wow, that was a while ago :-)

Ya, the wheels of change sometimes turn with glacial speed !

It would be nice to have these changes as the starting point for some
functionality improvements to make a minor version release (5.5.0) . . .

Though, given Arthur's experience with the tip of the trunk, we will
likely want to improve our coordination for cross platform compile and
sanity test runs.  Testing on a crusty netbsd machine,  the build fails
with the following,

    WARNING: `automake-1.9' is needed, and you do not seem to have it
    handy on your        
                           system.  You might have modified some files
    without having the
                           proper tools for further handling them.
    Check the `README' file,
                           it often tells you about the needed
    prerequirements for installing
                           this package.  You may also peek at any GNU
    archive site, in case
                           some other package would contain this missing
    `automake-1.9' program.



> If there is interest I'll gladly send in the changes I made, just let  
> me know.

Most certainly, if you can send diffs relative to the current CVS, that
would be nice.  With the stability of the codebase, it is likely that
this will work.  However, if there are conflicts, we may need to diff
with whatever version on which you had based the changes and consider
the most appropriate way to bring them forward.

more,
l8r,
v

--
"The future is here. It's just not evenly distributed yet."
 -- William Gibson, quoted by Whitfield Diffie


Reply | Threaded
Open this post in threaded view
|

Re: libwww contributions

Vic Bancroft-2
In reply to this post by Vic Bancroft-2

Arthur Smith wrote:

> Vic Bancroft wrote:
>
>> Arthur Smith wrote:
>>
>>> I had trouble compiling the latest (CVS) under solaris, anybody else
>>> tried this?
>>
>> Ouch, could you send a compile log ?
>
> Hi Vic - notice there are a bunch of 'regparm' warnings as well, but
> the error is with some dirent/direct stuff in HTFile.c:

Yea, I get the regparm warnings on a sparc machine running netbsd with
gcc version 2.95.3.  Of course that environment is so crusty, it dies in
some other strange way using libtool on md5.c . . .

Nevertheless, the way your build is failing seems directly related to
the introduction of the new function to calculate an appropriate name
buffer size in the dirent struct.

> This is on Sun Solaris 9, sparc (not x86) platform, with gcc 3.3 - do
> you want the full config.log?

Yes, if you do not mind, please send me the config log . . .

> make  all-recursive
> make[1]: Entering directory
> `/home/jis/resdev/apsmith/src/w3c-libwww-cvs/libwww'
> Making all in config
> make[2]: Entering directory
> `/home/jis/resdev/apsmith/src/w3c-libwww-cvs/libwww/config'
> make[2]: Nothing to be done for `all'.
> make[2]: Leaving directory
> `/home/jis/resdev/apsmith/src/w3c-libwww-cvs/libwww/config'
> Making all in modules
> make[2]: Entering directory
> `/home/jis/resdev/apsmith/src/w3c-libwww-cvs/libwww/modules'
> Making all in expat
> make[3]: Entering directory
> `/home/jis/resdev/apsmith/src/w3c-libwww-cvs/libwww/modules/expat'
> /bin/bash ./libtool --silent --mode=compile gcc -g -O2 -Wall
> -Wmissing-prototypes -Wstrict-prototypes -fexceptions   -I./lib -I. -o
> lib/xmlparse.lo -c lib/xmlparse.c
> In file included from lib/xmlparse.c:94:
> lib/xmltok.h:143: warning: `regparm' attribute directive ignored
> lib/xmltok.h:144: warning: `regparm' attribute directive ignored
>
This warning is expected behavior.  Since the regparm attribute, used on
i386 machines to pass arguments using registers, is unsave for global
functions in shared libraries on some ELF systems.

Perhaps we could detect the operating systems on which gcc will issue
the warning and #define PTRFASTCALL in modules/expat/lib/internal.h
appropriately . . .

> gcc -DHAVE_CONFIG_H -I. -I. -I../.. -I/usr/local/include
> -I../../modules/md5 -I../../modules/expat/lib
> -DW3C_ICONS=\"/home/jis/resdev/apsmith/share/w3c-libwww\" -g -O2 -Wall
> -MT HTBind.lo -MD -MP -MF .deps/HTBind.Tpo -c HTBind.c -o HTBind.o
> >/dev/null 2>&1
> if /bin/sh ../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H
> -I. -I. -I../.. -I/usr/local/include -I../../modules/md5
> -I../../modules/expat/lib
> -DW3C_ICONS=\"/home/jis/resdev/apsmith/share/w3c-libwww\"   -g -O2
> -Wall -MT HTFile.lo -MD -MP -MF ".deps/HTFile.Tpo" -c -o HTFile.lo
> HTFile.c; \
> then mv -f ".deps/HTFile.Tpo" ".deps/HTFile.Plo"; else rm -f
> ".deps/HTFile.Tpo"; exit 1; fi
> gcc -DHAVE_CONFIG_H -I. -I. -I../.. -I/usr/local/include
> -I../../modules/md5 -I../../modules/expat/lib
> -DW3C_ICONS=\"/home/jis/resdev/apsmith/share/w3c-libwww\" -g -O2 -Wall
> -MT HTFile.lo -MD -MP -MF .deps/HTFile.Tpo -c HTFile.c  -fPIC -DPIC -o
> .libs/HTFile.o
> HTFile.c:702:2: #error "buffer size for readdir_r cannot be determined"
> HTFile.c: In function `HTFile_dirent_buf_size':
> HTFile.c:705: warning: implicit declaration of function `offsetof'
> HTFile.c:705: error: parse error before "struct"
> make[5]: *** [HTFile.lo] Error 1
> make[5]: Leaving directory
> `/home/jis/resdev/apsmith/src/w3c-libwww-cvs/libwww/Library/src'
> make[4]: *** [all-recursive] Error 1
> make[4]: Leaving directory
> `/home/jis/resdev/apsmith/src/w3c-libwww-cvs/libwww/Library/src'
> make[3]: *** [all] Error 2
> make[3]: Leaving directory
> `/home/jis/resdev/apsmith/src/w3c-libwww-cvs/libwww/Library/src'
> make[2]: *** [all-recursive] Error 1
> make[2]: Leaving directory
> `/home/jis/resdev/apsmith/src/w3c-libwww-cvs/libwww/Library'
> make[1]: *** [all-recursive] Error 1
> make[1]: Leaving directory
> `/home/jis/resdev/apsmith/src/w3c-libwww-cvs/libwww'
> make: *** [all] Error 2

Ouch.  It does seem a tad harsh that just missing a NAME_MAX macro just
signals an error.

Perhaps Solaris 9 doesn't have all it's POSIX definitions sorted out the
way we are using it from /usr/include/dirent.h.

Does Solaris make the BSD equivalent MAXNAMLEN in
/usr/include/sys/dirent.h ?

Of course, we could be safer and use the FILENAME_MAX macro for the
stdio.h crowd, it is set to 4096 on my system.

In any case, to get past the compile error, you could try to replace
line 702 with

               name_max = 4096;

What do you think Ben ?

more,
l8r,
v

--
"The future is here. It's just not evenly distributed yet."
 -- William Gibson, quoted by Whitfield Diffie


Reply | Threaded
Open this post in threaded view
|

Re: libwww contributions

Ben Hutchings-2
Vic Bancroft wrote:
> Arthur Smith wrote:
<snip>

> > gcc -DHAVE_CONFIG_H -I. -I. -I../.. -I/usr/local/include
> > -I../../modules/md5 -I../../modules/expat/lib
> > -DW3C_ICONS=\"/home/jis/resdev/apsmith/share/w3c-libwww\" -g -O2 -Wall
> > -MT HTBind.lo -MD -MP -MF .deps/HTBind.Tpo -c HTBind.c -o HTBind.o
> > >/dev/null 2>&1
> > if /bin/sh ../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H
> > -I. -I. -I../.. -I/usr/local/include -I../../modules/md5
> > -I../../modules/expat/lib
> > -DW3C_ICONS=\"/home/jis/resdev/apsmith/share/w3c-libwww\"   -g -O2
> > -Wall -MT HTFile.lo -MD -MP -MF ".deps/HTFile.Tpo" -c -o HTFile.lo
> > HTFile.c; \
> > then mv -f ".deps/HTFile.Tpo" ".deps/HTFile.Plo"; else rm -f
> > ".deps/HTFile.Tpo"; exit 1; fi
> > gcc -DHAVE_CONFIG_H -I. -I. -I../.. -I/usr/local/include
> > -I../../modules/md5 -I../../modules/expat/lib
> > -DW3C_ICONS=\"/home/jis/resdev/apsmith/share/w3c-libwww\" -g -O2 -Wall
> > -MT HTFile.lo -MD -MP -MF .deps/HTFile.Tpo -c HTFile.c  -fPIC -DPIC -o
> > .libs/HTFile.o
> > HTFile.c:702:2: #error "buffer size for readdir_r cannot be determined"
> > HTFile.c: In function `HTFile_dirent_buf_size':
> > HTFile.c:705: warning: implicit declaration of function `offsetof'
<snip>

Please note the list of headers at the top of the sample code I gave.
On a POSIX-compliant system you should be including all of these:

#include <sys/types.h>
#include <dirent.h>
#include <limits.h>
#include <stddef.h>
#include <unistd.h>

It may be necessary to make some of them conditional on autoconf macros
in order to support older Unix variants.

> In any case, to get past the compile error, you could try to replace
> line 702 with
>
>                name_max = 4096;
>
> What do you think Ben ?

First, fix the real problem.  That may be useful as a fallback where the
limits really aren't available.

Ben.

--
Ben Hutchings
friends: People who know you well, but like you anyway.

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: libwww contributions

Vic Bancroft-2

Another email that did not seem to reach the lists.

Ben Hutchings wrote:

> Please note the list of headers at the top of the sample code I gave.
>
>On a POSIX-compliant system you should be including all of these:
>
>#include <sys/types.h>
>#include <dirent.h>
>#include <limits.h>
>#include <stddef.h>
>#include <unistd.h>
>
>It may be necessary to make some of them conditional on autoconf macros
>in order to support older Unix variants.
>  
>
Of the list the only missing one was stddef.h . . .

   Index: configure.ac
   ===================================================================
   RCS file: /sources/public/libwww/configure.ac,v
   retrieving revision 1.5
   diff -r1.5 configure.ac
   248a249
    > AC_CHECK_HEADERS(stddef.h)

We check for it having been detected with a modififcation to the
wwwsys.html,

   Index: Library/src/wwwsys.html
   ===================================================================
   RCS file: /sources/public/libwww/Library/src/wwwsys.html,v
   retrieving revision 2.127
   diff -r2.127 wwwsys.html
   747a748,752
    > /* stdef.h */
    > #ifdef HAVE_STDDEF_H
    > #include <stddef.h>
    > #endif
    >

Of course, one has to run autoheader to get the appropriate fragment
into wwwconf.h.in . . .

Does this fix the borken compile under Solaris ?  Does it break anything
else ?

In any case, here is the round of revisions,

 ChangeLog new revision: 1.54;
 configure.ac new revision: 1.6;
 wwwsys.html new revision: 2.128 ;

done.

more,
l8r,
v

--
"The future is here. It's just not evenly distributed yet."
 -- William Gibson, quoted by Whitfield Diffie