Re: 100 Continue and Expects

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Greg Wilkins

Hi,

I'd like to wake up the discussion about 100 Continues in the specification.

In my experience with the Jetty server, one of the main users of Expect
continues is .net clients doing webservices.   Unfortunately there appears
to be a some clients that although they send expect 100 continues, they
always send the body regardless if the server has sent 100 continues
or not.

This behaviour is kind of allowed by RFC2616 as it says:

  "Because of the presence of older implementations, the protocol allows
  ambiguous situations in which a client may send "Expect: 100- continue"
  without receiving either a 417 (Expectation Failed) status or a 100
  (Continue) status. Therefore, when a client sends this header field
  to an origin server (possibly via a proxy) from which it has never
  seen a 100 (Continue) status, the client SHOULD NOT wait for an
  indefinite period before sending the request body"

The issue with this, is what happens if the server does not
send a 100 continues on the very first request it receives
from a client with the Expect: 100 header.

The spec allows container to not-send a 100 continues if there
is already content available to be read:

  "  - An origin server MAY omit a 100 (Continue) response if it has
        already received some or all of the request body for the
        corresponding request."

This is fine if the container wants to read and handle that content.

However, the case that is not covered by the spec, is what if
the server does not want to read the body (eg it wants to send
a 401 or 302), but there is already content available to be read.

It should not be possible for this content to be a pipelined
request - as using pipelining with expect 100 cannot possibly
work (and maybe spec should explicitly say they can't be used
together).    So the data must be the body of the request
that was sent regardless by the client (maybe after NOT
waiting for an indefinite period).


So it would be great if the spec could give guidance to
tell servers what to do with data that is received
before any response is sent to an expect 100 continues.

Perhaps something like:

    An origin server MUST read the request body if it has
        already received some or all of the request body for the
        corresponding request before sending any response


There is still an implicit race here... but if clients are
going to send bodies regardless of the 100 continue mechanisms
then servers need to be able to read those bodies regardless
as well.


cheers





















Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

David Morris-4



On Sat, 27 Mar 2010, Greg Wilkins wrote:

> I'd like to wake up the discussion about 100 Continues in the specification.
>
> In my experience with the Jetty server, one of the main users of Expect
> continues is .net clients doing webservices.   Unfortunately there appears
> to be a some clients that although they send expect 100 continues, they
> always send the body regardless if the server has sent 100 continues
> or not.
>
...

>
> So it would be great if the spec could give guidance to
> tell servers what to do with data that is received
> before any response is sent to an expect 100 continues.
>
> Perhaps something like:
>
>     An origin server MUST read the request body if it has
>         already received some or all of the request body for the
>         corresponding request before sending any response

I don't think so ... the server MUST be prepared to deal with the
unexpected input ... could be a connection reset ... I don't think
it is our place to dictate error handling.

A new MUST also isn't possible as that would change the protocol.

Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Greg Wilkins
David Morris wrote:

>
>
> On Sat, 27 Mar 2010, Greg Wilkins wrote:
>
>> I'd like to wake up the discussion about 100 Continues in the specification.
>>
>> In my experience with the Jetty server, one of the main users of Expect
>> continues is .net clients doing webservices.   Unfortunately there appears
>> to be a some clients that although they send expect 100 continues, they
>> always send the body regardless if the server has sent 100 continues
>> or not.
>>
> ...
>> So it would be great if the spec could give guidance to
>> tell servers what to do with data that is received
>> before any response is sent to an expect 100 continues.
>>
>> Perhaps something like:
>>
>>     An origin server MUST read the request body if it has
>>         already received some or all of the request body for the
>>         corresponding request before sending any response
>
> I don't think so ... the server MUST be prepared to deal with the
> unexpected input ... could be a connection reset ... I don't think
> it is our place to dictate error handling.

David,

I'm not sure what you are saying here, as this is not really error
handling.   In the circumstances I'm describing, the spec
allows a client to either not send the body (because it never
got a 100 continues) or it can send a body (because it didn't
delay an unreasonable time).

But the spec is not clear on how the server should handle
the later case, when the server is actually a RFC2616 compliant
server.


> A new MUST also isn't possible as that would change the protocol.

Oops I admit I've not read the bis charter, so I didn't know that
new MUSTs  are not possible.    However, I don't really think this
is a change in the protocol, just some guidance on how to handle
a little ambiguous corner.  A MAY or SHOULD would be equally
helpful in reminding servers that clients may send bodies
even if 100 is not sent.


cheers


Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Jamie Lokier
Greg Wilkins wrote:
> new MUSTs  are not possible.    However, I don't really think this
> is a change in the protocol, just some guidance on how to handle
> a little ambiguous corner.  A MAY or SHOULD would be equally
> helpful in reminding servers that clients may send bodies
> even if 100 is not sent.

The client may send a request body, or close the connection.  What it
can't do is omit the body and send another request on the same
connection.  That said, if it's sending chunked, the client could
truncate the body it's sending if it believes the server doesn't care
about the body, and then reuse the connection.  Because nobody does
chunked requests, though, I'd be hesitant about assuming it works.

I found the section in RFC2616 about the client not having to send the
request body when it receives an error a bit confusing, but after
asking this list a couple of years ago, it was clarified that the
client may send it or close the connection, but it can't simply not
send a body and reuse the connection for another request (unfortunately).

The only benefit of 100 Continue is that the client can read the
response and close, without saturating the link with an unused request
body.

  => RFC2616 does refer to an "error" response in this case, which
     is also a bit confusing - because this behaviour should be fine with
     *any* response code, and it's useful for 3xx.  Perhaps this could
     be relaxed in HTTPbis?

I did experiment with streaming a response at the same time as sending
the request body - which should be fine with the abort rule as long as
the server sends 100 Continue first - but found most clients and
servers didn't behave well with mutual streaming like this.  A few
were ok, which would have been a nice way to reduce connections for
some ajaxy/comety applications.

-- Jamie


Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Greg Wilkins
Jamie,

thanks for that info.  I had not realized that the client had
not choice but to not reuse the connection if a non-100 response
is received.

Then perhaps the bis spec can enhanced to make this point clearer.

It might also be good to suggest that servers send Connection: close
with non 100 responses if expect 100 is received.  This would
make clients that don't know they should close, do the close.

cheers


Jamie Lokier wrote:

> Greg Wilkins wrote:
>> new MUSTs  are not possible.    However, I don't really think this
>> is a change in the protocol, just some guidance on how to handle
>> a little ambiguous corner.  A MAY or SHOULD would be equally
>> helpful in reminding servers that clients may send bodies
>> even if 100 is not sent.
>
> The client may send a request body, or close the connection.  What it
> can't do is omit the body and send another request on the same
> connection.  That said, if it's sending chunked, the client could
> truncate the body it's sending if it believes the server doesn't care
> about the body, and then reuse the connection.  Because nobody does
> chunked requests, though, I'd be hesitant about assuming it works.
>
> I found the section in RFC2616 about the client not having to send the
> request body when it receives an error a bit confusing, but after
> asking this list a couple of years ago, it was clarified that the
> client may send it or close the connection, but it can't simply not
> send a body and reuse the connection for another request (unfortunately).
>
> The only benefit of 100 Continue is that the client can read the
> response and close, without saturating the link with an unused request
> body.
>
>   => RFC2616 does refer to an "error" response in this case, which
>      is also a bit confusing - because this behaviour should be fine with
>      *any* response code, and it's useful for 3xx.  Perhaps this could
>      be relaxed in HTTPbis?
>
> I did experiment with streaming a response at the same time as sending
> the request body - which should be fine with the abort rule as long as
> the server sends 100 Continue first - but found most clients and
> servers didn't behave well with mutual streaming like this.  A few
> were ok, which would have been a nice way to reduce connections for
> some ajaxy/comety applications.
>
> -- Jamie
>


Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Jamie Lokier
Greg Wilkins wrote:
> thanks for that info.  I had not realized that the client had
> not choice but to not reuse the connection if a non-100 response
> is received.

I believe it's ok to reuse the connection if it posts a request body
matching the headers.  However:

> Then perhaps the bis spec can enhanced to make this point clearer.

I found this part of RFC2616 a little confusing:

    8.2.2 Monitoring Connections for Error Status Messages

       An HTTP/1.1 (or later) client sending a message-body SHOULD
       monitor the network connection for an error status while it is
       transmitting the request. If the client sees an error status,
       it SHOULD immediately cease transmitting the body. If the body
       is being sent using a "chunked" encoding (section 3.6), a zero
       length chunk and empty trailer MAY be used to prematurely mark
       the end of the message.  If the body was preceded by a
       Content-Length header, the client MUST close the connection.

As far I know, the client does not have to close the connection, if
it continues to transmit the rest of the body.

And if it does close, of course it shouldn't do so until it's seen the
whole response, not just the "error status".

Moreover, I wasn't sure what _exactly_ counts as an "error status", as
it's not a defined term, and 3xx (not an error) would seem to need the
same behaviour.  Or even 2xx, if the request body isn't relevant.
(Although I really like the idea of simultaneously streaming
request and response bodies).

> It might also be good to suggest that servers send Connection: close
> with non 100 responses if expect 100 is received.  This would
> make clients that don't know they should close, do the close.

I don't think it's necessary for them to close, despite the MUST above
;-) In other words, I parse the above RFC2616 extract with the MUST
clause conditional on the client performing the SHOULD and ceasing to
transmit the body - if it hasn't already finished that.

Actually I go a bit further:

With certain types (Microsoft) of authentication, the client must not
close the connection...

Here's a proposed alternative text.  It could be tidied a bit:

    8.2.2 Monitoring Connections and Aborting Request Bodies

       An HTTP/1.1 (or later) client sending a message-body SHOULD
       monitor the network connection for a response while it is
       transmitting the request.  If the client sees a 3xx, 4xx or 5xx
       status, it SHOULD immediately cease transmitting the body (by
       which we also mean not starting to transmit if it has not
       started), although it MAY continue transmitting the body if
       there is not much remaining to transmit, the latter being an
       implementation-defined heuristic.  All references to ceasing to
       transmit the body shall

       The client SHOULD also cease transmitting the body, subject to
       the above option to continue transmitting, if it sees a 2xx
       status which is not preceded by 100 (Continue) after it had
       sent the Expect request header field with the "100-continue"
       expectation.  This allows a server to indicate that it is still
       consuming the request body while it sends the response, or to
       indicate that it does not require the request body.

       When ceasing to transmit the body, if the body is being sent
       using a "chunked" encoding (section 3.6), a zero length chunk
       and empty trailer MAY be used to prematurely mark the end of
       the message, and connection reuse is possible.

       When ceasing to transmit a body which was preceded by a
       Content-Length header, the client SHOULD halt transmission and
       wait until it has received the response, and it MUST close the
       connection without sending another request.  If the client
       requires the whole response, it MUST NOT signal end of
       transmission prior to receiving the whole response by
       "half-closing" the connection, in the sense described as
       simplex close in [RFC793] section 3.5, as that typically causes
       the server to cease transmitting the response.

An idea: When aborting with a zero length trunk, could we append an
"aborted" chunk-extension, to indicate clearly that it was truncated,
not ended cleanly?

-- Jamie

Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Henrik Nordström
In reply to this post by Greg Wilkins
lör 2010-03-27 klockan 10:24 -0700 skrev Greg Wilkins:

> The issue with this, is what happens if the server does not
> send a 100 continues on the very first request it receives
> from a client with the Expect: 100 header.

The server should send an 100 Continue when it accepts to receive the
body. Exception being if part of the body has already received by the
server.

> The spec allows container to not-send a 100 continues if there
> is already content available to be read:

Yes as it then knows the client is no longer waiting for 100 Continue.

> This is fine if the container wants to read and handle that content.

Irrelevant.

> However, the case that is not covered by the spec, is what if
> the server does not want to read the body (eg it wants to send
> a 401 or 302), but there is already content available to be read.

That case is the same as when there is no Expect: 100-continue. The
server then responds with the final status, and eats the request body if
it wants to keep the connection persistent. Or if it wants to close the
connection then waits until the response data has been ACK:ed at the TCP
level plus some more to give the client some chance to pick up the
response before connection is closed.

The last part is one of the reasons why 100 Continue was added, adding
structure in how servers can reject request bodies by closing the
connection in controlled manners.

But it's true the wording in RFC2616 leaves much of the detail out or
hides it scattered over many different and seemingly unrelated sections.

> It should not be possible for this content to be a pipelined
> request - as using pipelining with expect 100 cannot possibly
> work (and maybe spec should explicitly say they can't be used
> together).    So the data must be the body of the request
> that was sent regardless by the client (maybe after NOT
> waiting for an indefinite period).

Correct.

> So it would be great if the spec could give guidance to
> tell servers what to do with data that is received
> before any response is sent to an expect 100 continues.
>
> Perhaps something like:
>
>     An origin server MUST read the request body if it has
>         already received some or all of the request body for the
>         corresponding request before sending any response

No.

> There is still an implicit race here... but if clients are
> going to send bodies regardless of the 100 continue mechanisms
> then servers need to be able to read those bodies regardless
> as well.

>From 100 Continue server requirements:

server handling of request with 100-continue expectation:

        If it responds with a final status
        code, it MAY close the transport connection or it MAY continue
        to read and discard the rest of the request.

server handling when there is no such expectation:

        request includes a request body, and the server responds
        with a final status code before reading the entire request body
        from the transport connection, then the server SHOULD NOT close
        the transport connection until it has read the entire request,
        or until the client closes the connection.


The race window you worry about is covered in the section following this
"Client Behavior if Server Prematurely Closes Connection", which
squarely puts the burden on the client if it's not being careful when
sending request bodies..

But in real-life and experience servers need to be careful as well, and
is reflected in some other sections of the specifications requiring
servers to perform a graceful close and/or wait for TCP ACK of the
response data.

Regards
Henrik


Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Henrik Nordström
In reply to this post by Jamie Lokier
sön 2010-03-28 klockan 15:20 +0100 skrev Jamie Lokier:
>
> With certain types (Microsoft) of authentication

You forgot to add "which is not true HTTP authentication schemes". That
family of auth schemes makes many assumptions which is opposite the
intentions of the HTTP specifications so using them as an example when
trying to understand the wording of the specification text is not valid.

Still interesting when talking about what should be said as they are a
reality and something HTTP has to deal with today, but not when trying
to understand why RFC2616 is written in a certain manner.

Regards
Henrik


Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Mark Nottingham-4
In reply to this post by Jamie Lokier

On 28/03/2010, at 7:20 AM, Jamie Lokier wrote:

>
> I found this part of RFC2616 a little confusing:
>
>    8.2.2 Monitoring Connections for Error Status Messages
>
>       An HTTP/1.1 (or later) client sending a message-body SHOULD
>       monitor the network connection for an error status while it is
>       transmitting the request. If the client sees an error status,
>       it SHOULD immediately cease transmitting the body. If the body
>       is being sent using a "chunked" encoding (section 3.6), a zero
>       length chunk and empty trailer MAY be used to prematurely mark
>       the end of the message.  If the body was preceded by a
>       Content-Length header, the client MUST close the connection.
>
> As far I know, the client does not have to close the connection, if
> it continues to transmit the rest of the body.
>
> And if it does close, of course it shouldn't do so until it's seen the
> whole response, not just the "error status".
>
> Moreover, I wasn't sure what _exactly_ counts as an "error status", as
> it's not a defined term, and 3xx (not an error) would seem to need the
> same behaviour.  Or even 2xx, if the request body isn't relevant.
> (Although I really like the idea of simultaneously streaming
> request and response bodies).


See <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/108>.

Cheers,

--
Mark Nottingham       [hidden email]



Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Jamie Lokier
In reply to this post by Henrik Nordström
Henrik Nordström wrote:
> But in real-life and experience servers need to be careful as well, and
> is reflected in some other sections of the specifications requiring
> servers to perform a graceful close and/or wait for TCP ACK of the
> response data.

I think it's a good idea to combine graceful close with TCP ACK when
possible, and otherwise to do graceful close.  But not TCP ACK alone,
as suggested by your "and/or".

I would caution against depending on TCP ACK *alone*, because it fails
in three scenarios:

  1. Sometimes it's HTTPS relayed over a proxy using CONNECT.
     Then TCP ACK does not indicate the client TCP has received the data.
 
  2. Many client TCPs will discard received data on reception of the
     TCP RST from the closed server TCP, before the client application
     receives the data from its TCP.

  3. Sometimes data is relayed over a TCP retransmitter.
     Then TCP ACK does not indicate the client TCP has received the data.

     I've read that some networks do this for all connections, to
     compensate for channel characteristics (radio).  But I've only
     seen it used for certain kinds of tunnelling, and to bypass firewalls.

-- Jamie

Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Jamie Lokier
In reply to this post by Henrik Nordström
Henrik Nordström wrote:

> sön 2010-03-28 klockan 15:20 +0100 skrev Jamie Lokier:
> >
> > With certain types (Microsoft) of authentication
>
> You forgot to add "which is not true HTTP authentication schemes". That
> family of auth schemes makes many assumptions which is opposite the
> intentions of the HTTP specifications so using them as an example when
> trying to understand the wording of the specification text is not valid.
>
> Still interesting when talking about what should be said as they are a
> reality and something HTTP has to deal with today, but not when trying
> to understand why RFC2616 is written in a certain manner.

Agreed, Microsoft's version is not standard HTTP.

Regardless of why, it's important to recognise that clients have the
option to send the whole request body if they decide to, despite one
possible reading of that section of RFC2616 that they must abort.

-- Jamie

Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Adrien de Croy

I did a review of what browsers do in some cases like this (e.g POST +
auth/NTLM) a while back.

Their behaviour varies.  For instance IE (even current) if it thinks it
will get an auth challenge, will submit initial POSTs with a
content-length 0 (this breaks a heap of stuff).

FF and Chrome will send the whole POST body as many times as required to
get through auth.

Server behaviour varies also.  IIS5 for instance will send a 100
continue on all POST requests regardless of whether they contain
Expects.  In fact I've never seen a browser send Expects.

I wrote a I-D partly about this a while back, but it was somewhat
misguided, based on my misunderstanding of the Expects mechanism.

In the end, I think that's the biggest problem with Expects - people
expect (pardon the pun) more from it than it delivers.  The reason the
history of this list contains numerous discussions about Expects, is
because of common misinterpretation.

The wording in the RFC implies that Expects and 100 Continue can be used
to avoid sending the body of a request.  It omits to mention that if
indeed you do want to avoid this (e.g. when you get a 3xx or 4xx), you
either need to disconnect and re-try, or complete the message some other
way (e.g. use chunking and prematurely complete it with a 0 chunk).

I would really hate to see anyone adopt the 0 chunk approach - it's just
as bad as sending Content-Length: 0 when the browser thinks the proxy or
server will send an auth challenge.  Browsers second-guessing proxies /
servers is not a good option, and when the browser gets it wrong, things
get ugly.  We had to write a hack into WinGate to cope with the IE
misbehaviour (WinGate only sends an auth challenge when policy dictates
auth is required - the browser cannot possibly predict this, and a 0
length post is valid, but breaks websites).  Sending a 0 length chunk
tells the proxy the message is complete.  It could then go through many
stages of processing that is really undesirable.  There's no way to
signal an abort and maintain a connection.

In the end, the best option IMO is to send the whole thing each time (as
per FF and Chrome - sorry don't know what Opera does).  It can be
hideously painful with chained connection-oriented auth over a slow link.

So actually there's no clean solution to this problem.  I proposed a
while back a kind of HEAD command for POST to establish a path with
credentials before posting a body.  But in the end, any proper solution
to this problem will be a significant protocol change to HTTP (since
HTTP is designed for intermediaries to be able to work with unknown
methods), which means we'll be living with this problem for a long time yet.

As for NTLM, it's not going away either.  So much as people don't like
it because it's not HTTP compliant auth - it's the reality we are stuck
with.

Cheers

Adrien


On 30/03/2010 12:22 p.m., Jamie Lokier wrote:

> Henrik Nordström wrote:
>    
>> sön 2010-03-28 klockan 15:20 +0100 skrev Jamie Lokier:
>>      
>>> With certain types (Microsoft) of authentication
>>>        
>> You forgot to add "which is not true HTTP authentication schemes". That
>> family of auth schemes makes many assumptions which is opposite the
>> intentions of the HTTP specifications so using them as an example when
>> trying to understand the wording of the specification text is not valid.
>>
>> Still interesting when talking about what should be said as they are a
>> reality and something HTTP has to deal with today, but not when trying
>> to understand why RFC2616 is written in a certain manner.
>>      
> Agreed, Microsoft's version is not standard HTTP.
>
> Regardless of why, it's important to recognise that clients have the
> option to send the whole request body if they decide to, despite one
> possible reading of that section of RFC2616 that they must abort.
>
> -- Jamie
>
>    

Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Yngve Nysaeter Pettersen
On Tue, 30 Mar 2010 03:42:16 +0300, Adrien de Croy <[hidden email]> wrote:

> In fact I've never seen a browser send Expects.

Opera will send "Expect: 100-continue" if the body is >=10 KB, and will in  
such cases wait 1.5s before sending the body.

--
Sincerely,
Yngve N. Pettersen

********************************************************************
Senior Developer                     Email: [hidden email]
Opera Software ASA                   http://www.opera.com/
Phone:  +47 24 16 42 60              Fax:    +47 24 16 40 01
********************************************************************

Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Mark Pauley
In reply to this post by Adrien de Croy
CFNetwork on Mac OSX will send a zero-length post as the first leg of the 3-way auth for NTLM after it has received the initial 4xx response.

CFNetwork ought to soon allow clients to specify Expects and handle the 100 Continue properly to reduce bandwidth on any requests with a large payload, though I can't say when 'soon' will be.  Unfortunately, the NTLM case really is dictated by Microsoft.  Their own behavior shows that they believe it's okay to send a POST with a zero length for the initial NTLM transaction, and since Microsoft is by far the largest producer of NTLM-using server software, CFNetwork just ape's IE.  What about the zero-length post breaks HTTP?  It seems like a POST that both has length 0 and Expects: 100-continue is silly, and the client ought to be able to handle either a 100-continue followed by standard HTTP response or simply just a standard HTTP response.  Garbage in, garbage out.


On Mar 29, 2010, at 5:42 PM, Adrien de Croy wrote:

>
> I did a review of what browsers do in some cases like this (e.g POST + auth/NTLM) a while back.
>
> Their behaviour varies.  For instance IE (even current) if it thinks it will get an auth challenge, will submit initial POSTs with a content-length 0 (this breaks a heap of stuff).
>
> FF and Chrome will send the whole POST body as many times as required to get through auth.
>
> Server behaviour varies also.  IIS5 for instance will send a 100 continue on all POST requests regardless of whether they contain Expects.  In fact I've never seen a browser send Expects.
>
> I wrote a I-D partly about this a while back, but it was somewhat misguided, based on my misunderstanding of the Expects mechanism.
>
> In the end, I think that's the biggest problem with Expects - people expect (pardon the pun) more from it than it delivers.  The reason the history of this list contains numerous discussions about Expects, is because of common misinterpretation.
>
> The wording in the RFC implies that Expects and 100 Continue can be used to avoid sending the body of a request.  It omits to mention that if indeed you do want to avoid this (e.g. when you get a 3xx or 4xx), you either need to disconnect and re-try, or complete the message some other way (e.g. use chunking and prematurely complete it with a 0 chunk).
>
> I would really hate to see anyone adopt the 0 chunk approach - it's just as bad as sending Content-Length: 0 when the browser thinks the proxy or server will send an auth challenge.  Browsers second-guessing proxies / servers is not a good option, and when the browser gets it wrong, things get ugly.  We had to write a hack into WinGate to cope with the IE misbehaviour (WinGate only sends an auth challenge when policy dictates auth is required - the browser cannot possibly predict this, and a 0 length post is valid, but breaks websites).  Sending a 0 length chunk tells the proxy the message is complete.  It could then go through many stages of processing that is really undesirable.  There's no way to signal an abort and maintain a connection.
>
> In the end, the best option IMO is to send the whole thing each time (as per FF and Chrome - sorry don't know what Opera does).  It can be hideously painful with chained connection-oriented auth over a slow link.
>
> So actually there's no clean solution to this problem.  I proposed a while back a kind of HEAD command for POST to establish a path with credentials before posting a body.  But in the end, any proper solution to this problem will be a significant protocol change to HTTP (since HTTP is designed for intermediaries to be able to work with unknown methods), which means we'll be living with this problem for a long time yet.
>
> As for NTLM, it's not going away either.  So much as people don't like it because it's not HTTP compliant auth - it's the reality we are stuck with.
>
> Cheers
>
> Adrien
>
>
> On 30/03/2010 12:22 p.m., Jamie Lokier wrote:
>> Henrik Nordström wrote:
>>  
>>> sön 2010-03-28 klockan 15:20 +0100 skrev Jamie Lokier:
>>>    
>>>> With certain types (Microsoft) of authentication
>>>>      
>>> You forgot to add "which is not true HTTP authentication schemes". That
>>> family of auth schemes makes many assumptions which is opposite the
>>> intentions of the HTTP specifications so using them as an example when
>>> trying to understand the wording of the specification text is not valid.
>>>
>>> Still interesting when talking about what should be said as they are a
>>> reality and something HTTP has to deal with today, but not when trying
>>> to understand why RFC2616 is written in a certain manner.
>>>    
>> Agreed, Microsoft's version is not standard HTTP.
>>
>> Regardless of why, it's important to recognise that clients have the
>> option to send the whole request body if they decide to, despite one
>> possible reading of that section of RFC2616 that they must abort.
>>
>> -- Jamie
>>
>>  
>


Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Adrien de Croy

we saw the issue with IE + WinGate Proxy + NTLM going through to a
webmail site.

WinGate was configured to require auth, but credentials get inherited,
and policy can be used to specify which sites need auth or not.  So not
every connection was being asked to auth (407).

IE was obviously working on the assumption that it would ALWAYS be
challenged with a 407 on new connections.

Customers complained that they couldn't reliably post messages, the
webmail would randomly break.  We tracked it down to the 0 length POST
on a new connection.  IE thought it would get a 407, but it didn't, and
the proxy passed the 0 length post through to the webmail server, which
barfed on it, with the side-effect of the browser losing the email the
customer had just written.

I complained about this to MS, but I think they still think it's a good
idea.

Our hack fix was to check UA, 0 length etc, and if we get one,
arbitrarily send a 407 back.  It's really ugly and I wish to death that
MS would pull this broken behaviour.

Part of our selling point is that people can have policy which allows
for access to some sites to be anonymous, other sites need auth etc -
the admin chooses when and what needs auth.  There's no way the browser
can predict it will get a 407 or not, so to send a 0-length POST is
suicidal.

On 1/04/2010 11:29 a.m., Mark Pauley wrote:

> CFNetwork on Mac OSX will send a zero-length post as the first leg of the 3-way auth for NTLM after it has received the initial 4xx response.
>
> CFNetwork ought to soon allow clients to specify Expects and handle the 100 Continue properly to reduce bandwidth on any requests with a large payload, though I can't say when 'soon' will be.  Unfortunately, the NTLM case really is dictated by Microsoft.  Their own behavior shows that they believe it's okay to send a POST with a zero length for the initial NTLM transaction, and since Microsoft is by far the largest producer of NTLM-using server software, CFNetwork just ape's IE.  What about the zero-length post breaks HTTP?  It seems like a POST that both has length 0 and Expects: 100-continue is silly, and the client ought to be able to handle either a 100-continue followed by standard HTTP response or simply just a standard HTTP response.  Garbage in, garbage out.
>
>
> On Mar 29, 2010, at 5:42 PM, Adrien de Croy wrote:
>
>    
>> I did a review of what browsers do in some cases like this (e.g POST + auth/NTLM) a while back.
>>
>> Their behaviour varies.  For instance IE (even current) if it thinks it will get an auth challenge, will submit initial POSTs with a content-length 0 (this breaks a heap of stuff).
>>
>> FF and Chrome will send the whole POST body as many times as required to get through auth.
>>
>> Server behaviour varies also.  IIS5 for instance will send a 100 continue on all POST requests regardless of whether they contain Expects.  In fact I've never seen a browser send Expects.
>>
>> I wrote a I-D partly about this a while back, but it was somewhat misguided, based on my misunderstanding of the Expects mechanism.
>>
>> In the end, I think that's the biggest problem with Expects - people expect (pardon the pun) more from it than it delivers.  The reason the history of this list contains numerous discussions about Expects, is because of common misinterpretation.
>>
>> The wording in the RFC implies that Expects and 100 Continue can be used to avoid sending the body of a request.  It omits to mention that if indeed you do want to avoid this (e.g. when you get a 3xx or 4xx), you either need to disconnect and re-try, or complete the message some other way (e.g. use chunking and prematurely complete it with a 0 chunk).
>>
>> I would really hate to see anyone adopt the 0 chunk approach - it's just as bad as sending Content-Length: 0 when the browser thinks the proxy or server will send an auth challenge.  Browsers second-guessing proxies / servers is not a good option, and when the browser gets it wrong, things get ugly.  We had to write a hack into WinGate to cope with the IE misbehaviour (WinGate only sends an auth challenge when policy dictates auth is required - the browser cannot possibly predict this, and a 0 length post is valid, but breaks websites).  Sending a 0 length chunk tells the proxy the message is complete.  It could then go through many stages of processing that is really undesirable.  There's no way to signal an abort and maintain a connection.
>>
>> In the end, the best option IMO is to send the whole thing each time (as per FF and Chrome - sorry don't know what Opera does).  It can be hideously painful with chained connection-oriented auth over a slow link.
>>
>> So actually there's no clean solution to this problem.  I proposed a while back a kind of HEAD command for POST to establish a path with credentials before posting a body.  But in the end, any proper solution to this problem will be a significant protocol change to HTTP (since HTTP is designed for intermediaries to be able to work with unknown methods), which means we'll be living with this problem for a long time yet.
>>
>> As for NTLM, it's not going away either.  So much as people don't like it because it's not HTTP compliant auth - it's the reality we are stuck with.
>>
>> Cheers
>>
>> Adrien
>>
>>
>> On 30/03/2010 12:22 p.m., Jamie Lokier wrote:
>>      
>>> Henrik Nordström wrote:
>>>
>>>        
>>>> sön 2010-03-28 klockan 15:20 +0100 skrev Jamie Lokier:
>>>>
>>>>          
>>>>> With certain types (Microsoft) of authentication
>>>>>
>>>>>            
>>>> You forgot to add "which is not true HTTP authentication schemes". That
>>>> family of auth schemes makes many assumptions which is opposite the
>>>> intentions of the HTTP specifications so using them as an example when
>>>> trying to understand the wording of the specification text is not valid.
>>>>
>>>> Still interesting when talking about what should be said as they are a
>>>> reality and something HTTP has to deal with today, but not when trying
>>>> to understand why RFC2616 is written in a certain manner.
>>>>
>>>>          
>>> Agreed, Microsoft's version is not standard HTTP.
>>>
>>> Regardless of why, it's important to recognise that clients have the
>>> option to send the whole request body if they decide to, despite one
>>> possible reading of that section of RFC2616 that they must abort.
>>>
>>> -- Jamie
>>>
>>>
>>>        
>>      
>    

Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Mark Pauley
You're right, and we've identified these sort of issues as well.  CFNetwork (and any HTTP client in general) can't assume that they'll be given a 407, even if we know that this host has sent us one before.  Moreover, using a HEAD to 'prime' the auth also doesn't work, nor does anything similar work.  Sometimes you have actually try the full request, because HTTP allows for servers to be as pernicious as they please.

In the end, the only time CFNetwork zeroes out the POST's length is in the case that we've already gotten a 4xx, and we are starting the NTLM auth dance, because there is no way that NTLM can just decide to not give us a 4xx once we give it the NTLM initiation in response to the initial 4xx.


On Mar 31, 2010, at 3:48 PM, Adrien de Croy wrote:

>
> we saw the issue with IE + WinGate Proxy + NTLM going through to a webmail site.
>
> WinGate was configured to require auth, but credentials get inherited, and policy can be used to specify which sites need auth or not.  So not every connection was being asked to auth (407).
>
> IE was obviously working on the assumption that it would ALWAYS be challenged with a 407 on new connections.
>
> Customers complained that they couldn't reliably post messages, the webmail would randomly break.  We tracked it down to the 0 length POST on a new connection.  IE thought it would get a 407, but it didn't, and the proxy passed the 0 length post through to the webmail server, which barfed on it, with the side-effect of the browser losing the email the customer had just written.
>
> I complained about this to MS, but I think they still think it's a good idea.
>
> Our hack fix was to check UA, 0 length etc, and if we get one, arbitrarily send a 407 back.  It's really ugly and I wish to death that MS would pull this broken behaviour.
>
> Part of our selling point is that people can have policy which allows for access to some sites to be anonymous, other sites need auth etc - the admin chooses when and what needs auth.  There's no way the browser can predict it will get a 407 or not, so to send a 0-length POST is suicidal.
>
> On 1/04/2010 11:29 a.m., Mark Pauley wrote:
>> CFNetwork on Mac OSX will send a zero-length post as the first leg of the 3-way auth for NTLM after it has received the initial 4xx response.
>>
>> CFNetwork ought to soon allow clients to specify Expects and handle the 100 Continue properly to reduce bandwidth on any requests with a large payload, though I can't say when 'soon' will be.  Unfortunately, the NTLM case really is dictated by Microsoft.  Their own behavior shows that they believe it's okay to send a POST with a zero length for the initial NTLM transaction, and since Microsoft is by far the largest producer of NTLM-using server software, CFNetwork just ape's IE.  What about the zero-length post breaks HTTP?  It seems like a POST that both has length 0 and Expects: 100-continue is silly, and the client ought to be able to handle either a 100-continue followed by standard HTTP response or simply just a standard HTTP response.  Garbage in, garbage out.
>>
>>
>> On Mar 29, 2010, at 5:42 PM, Adrien de Croy wrote:
>>
>>  
>>> I did a review of what browsers do in some cases like this (e.g POST + auth/NTLM) a while back.
>>>
>>> Their behaviour varies.  For instance IE (even current) if it thinks it will get an auth challenge, will submit initial POSTs with a content-length 0 (this breaks a heap of stuff).
>>>
>>> FF and Chrome will send the whole POST body as many times as required to get through auth.
>>>
>>> Server behaviour varies also.  IIS5 for instance will send a 100 continue on all POST requests regardless of whether they contain Expects.  In fact I've never seen a browser send Expects.
>>>
>>> I wrote a I-D partly about this a while back, but it was somewhat misguided, based on my misunderstanding of the Expects mechanism.
>>>
>>> In the end, I think that's the biggest problem with Expects - people expect (pardon the pun) more from it than it delivers.  The reason the history of this list contains numerous discussions about Expects, is because of common misinterpretation.
>>>
>>> The wording in the RFC implies that Expects and 100 Continue can be used to avoid sending the body of a request.  It omits to mention that if indeed you do want to avoid this (e.g. when you get a 3xx or 4xx), you either need to disconnect and re-try, or complete the message some other way (e.g. use chunking and prematurely complete it with a 0 chunk).
>>>
>>> I would really hate to see anyone adopt the 0 chunk approach - it's just as bad as sending Content-Length: 0 when the browser thinks the proxy or server will send an auth challenge.  Browsers second-guessing proxies / servers is not a good option, and when the browser gets it wrong, things get ugly.  We had to write a hack into WinGate to cope with the IE misbehaviour (WinGate only sends an auth challenge when policy dictates auth is required - the browser cannot possibly predict this, and a 0 length post is valid, but breaks websites).  Sending a 0 length chunk tells the proxy the message is complete.  It could then go through many stages of processing that is really undesirable.  There's no way to signal an abort and maintain a connection.
>>>
>>> In the end, the best option IMO is to send the whole thing each time (as per FF and Chrome - sorry don't know what Opera does).  It can be hideously painful with chained connection-oriented auth over a slow link.
>>>
>>> So actually there's no clean solution to this problem.  I proposed a while back a kind of HEAD command for POST to establish a path with credentials before posting a body.  But in the end, any proper solution to this problem will be a significant protocol change to HTTP (since HTTP is designed for intermediaries to be able to work with unknown methods), which means we'll be living with this problem for a long time yet.
>>>
>>> As for NTLM, it's not going away either.  So much as people don't like it because it's not HTTP compliant auth - it's the reality we are stuck with.
>>>
>>> Cheers
>>>
>>> Adrien
>>>
>>>
>>> On 30/03/2010 12:22 p.m., Jamie Lokier wrote:
>>>    
>>>> Henrik Nordström wrote:
>>>>
>>>>      
>>>>> sön 2010-03-28 klockan 15:20 +0100 skrev Jamie Lokier:
>>>>>
>>>>>        
>>>>>> With certain types (Microsoft) of authentication
>>>>>>
>>>>>>          
>>>>> You forgot to add "which is not true HTTP authentication schemes". That
>>>>> family of auth schemes makes many assumptions which is opposite the
>>>>> intentions of the HTTP specifications so using them as an example when
>>>>> trying to understand the wording of the specification text is not valid.
>>>>>
>>>>> Still interesting when talking about what should be said as they are a
>>>>> reality and something HTTP has to deal with today, but not when trying
>>>>> to understand why RFC2616 is written in a certain manner.
>>>>>
>>>>>        
>>>> Agreed, Microsoft's version is not standard HTTP.
>>>>
>>>> Regardless of why, it's important to recognise that clients have the
>>>> option to send the whole request body if they decide to, despite one
>>>> possible reading of that section of RFC2616 that they must abort.
>>>>
>>>> -- Jamie
>>>>
>>>>
>>>>      
>>>    
>>  
>


Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Jamie Lokier
Mark Pauley wrote:
> You're right, and we've identified these sort of issues as well.
> CFNetwork (and any HTTP client in general) can't assume that they'll
> be given a 407, even if we know that this host has sent us one
> before.  Moreover, using a HEAD to 'prime' the auth also doesn't
> work, nor does anything similar work.  Sometimes you have actually
> try the full request, because HTTP allows for servers to be as
> pernicious as they please.

It's good to know that HEAD doesn't work either, thanks.

In theory a chunked request would be good for this, but I doubt if it works.

> In the end, the only time CFNetwork zeroes out the POST's length is
> in the case that we've already gotten a 4xx, and we are starting the
> NTLM auth dance, because there is no way that NTLM can just decide to
> not give us a 4xx once we give it the NTLM initiation in response to
> the initial 4xx.

So the result is you POST the request body twice - once before the
4xx, and once again at the last request?

What happens with combinations of NTLM proxy authentication and
"sometimes" NTLM server authentication, where you have to do the dance
for the proxy, then after that you may or may not have to do it a
second time?  Do you end up sending the request body exactly three
times in the case where NTLM is used for both hops?

Thanks,
-- Jamie

Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Adrien de Croy
In reply to this post by Mark Pauley

keep in mind also, it's possible on a single TCP connection to be
challenged for credentials multiple times.

It's a common customer requirement to enable some sites for some users,
and other sites for other users.  To establish who a user is requires auth.

A not too uncommon scenario through a proxy follows:

1. a user connects, requests a site
2. site not restricted, so request granted
3. on same connection, user requests a restricted site which only
members of group A are allowed to see
4. Proxy sends 407
5. user fills in credentials of a user in that group, request is granted
6. on same connection, user requests a restricted site which only
members of group B are allowed to see, and the user they logged in as
isn't a member
7. Proxy sends 407

Also if a user chooses credentials that don't grant the right to view
the page, a common response is another 407 - give the user another
chance.  Many browsers will give up after a few tries.

Proxy policy for what needs to be authed can vary on any parameter (even
method, or a header or even time of day).  I had a customer yesterday
who wanted to block POSTs to a specific site unless authed but only
during work hours.

So there's no guarantee that ANY request which is not the actual request
you wish to make will elicit the same response from a proxy, or be
reliably usable to pre-establish necessary credentials.

Adrien

On 1/04/2010 11:59 a.m., Mark Pauley wrote:

> You're right, and we've identified these sort of issues as well.  CFNetwork (and any HTTP client in general) can't assume that they'll be given a 407, even if we know that this host has sent us one before.  Moreover, using a HEAD to 'prime' the auth also doesn't work, nor does anything similar work.  Sometimes you have actually try the full request, because HTTP allows for servers to be as pernicious as they please.
>
> In the end, the only time CFNetwork zeroes out the POST's length is in the case that we've already gotten a 4xx, and we are starting the NTLM auth dance, because there is no way that NTLM can just decide to not give us a 4xx once we give it the NTLM initiation in response to the initial 4xx.
>
>
> On Mar 31, 2010, at 3:48 PM, Adrien de Croy wrote:
>
>    
>> we saw the issue with IE + WinGate Proxy + NTLM going through to a webmail site.
>>
>> WinGate was configured to require auth, but credentials get inherited, and policy can be used to specify which sites need auth or not.  So not every connection was being asked to auth (407).
>>
>> IE was obviously working on the assumption that it would ALWAYS be challenged with a 407 on new connections.
>>
>> Customers complained that they couldn't reliably post messages, the webmail would randomly break.  We tracked it down to the 0 length POST on a new connection.  IE thought it would get a 407, but it didn't, and the proxy passed the 0 length post through to the webmail server, which barfed on it, with the side-effect of the browser losing the email the customer had just written.
>>
>> I complained about this to MS, but I think they still think it's a good idea.
>>
>> Our hack fix was to check UA, 0 length etc, and if we get one, arbitrarily send a 407 back.  It's really ugly and I wish to death that MS would pull this broken behaviour.
>>
>> Part of our selling point is that people can have policy which allows for access to some sites to be anonymous, other sites need auth etc - the admin chooses when and what needs auth.  There's no way the browser can predict it will get a 407 or not, so to send a 0-length POST is suicidal.
>>
>> On 1/04/2010 11:29 a.m., Mark Pauley wrote:
>>      
>>> CFNetwork on Mac OSX will send a zero-length post as the first leg of the 3-way auth for NTLM after it has received the initial 4xx response.
>>>
>>> CFNetwork ought to soon allow clients to specify Expects and handle the 100 Continue properly to reduce bandwidth on any requests with a large payload, though I can't say when 'soon' will be.  Unfortunately, the NTLM case really is dictated by Microsoft.  Their own behavior shows that they believe it's okay to send a POST with a zero length for the initial NTLM transaction, and since Microsoft is by far the largest producer of NTLM-using server software, CFNetwork just ape's IE.  What about the zero-length post breaks HTTP?  It seems like a POST that both has length 0 and Expects: 100-continue is silly, and the client ought to be able to handle either a 100-continue followed by standard HTTP response or simply just a standard HTTP response.  Garbage in, garbage out.
>>>
>>>
>>> On Mar 29, 2010, at 5:42 PM, Adrien de Croy wrote:
>>>
>>>
>>>        
>>>> I did a review of what browsers do in some cases like this (e.g POST + auth/NTLM) a while back.
>>>>
>>>> Their behaviour varies.  For instance IE (even current) if it thinks it will get an auth challenge, will submit initial POSTs with a content-length 0 (this breaks a heap of stuff).
>>>>
>>>> FF and Chrome will send the whole POST body as many times as required to get through auth.
>>>>
>>>> Server behaviour varies also.  IIS5 for instance will send a 100 continue on all POST requests regardless of whether they contain Expects.  In fact I've never seen a browser send Expects.
>>>>
>>>> I wrote a I-D partly about this a while back, but it was somewhat misguided, based on my misunderstanding of the Expects mechanism.
>>>>
>>>> In the end, I think that's the biggest problem with Expects - people expect (pardon the pun) more from it than it delivers.  The reason the history of this list contains numerous discussions about Expects, is because of common misinterpretation.
>>>>
>>>> The wording in the RFC implies that Expects and 100 Continue can be used to avoid sending the body of a request.  It omits to mention that if indeed you do want to avoid this (e.g. when you get a 3xx or 4xx), you either need to disconnect and re-try, or complete the message some other way (e.g. use chunking and prematurely complete it with a 0 chunk).
>>>>
>>>> I would really hate to see anyone adopt the 0 chunk approach - it's just as bad as sending Content-Length: 0 when the browser thinks the proxy or server will send an auth challenge.  Browsers second-guessing proxies / servers is not a good option, and when the browser gets it wrong, things get ugly.  We had to write a hack into WinGate to cope with the IE misbehaviour (WinGate only sends an auth challenge when policy dictates auth is required - the browser cannot possibly predict this, and a 0 length post is valid, but breaks websites).  Sending a 0 length chunk tells the proxy the message is complete.  It could then go through many stages of processing that is really undesirable.  There's no way to signal an abort and maintain a connection.
>>>>
>>>> In the end, the best option IMO is to send the whole thing each time (as per FF and Chrome - sorry don't know what Opera does).  It can be hideously painful with chained connection-oriented auth over a slow link.
>>>>
>>>> So actually there's no clean solution to this problem.  I proposed a while back a kind of HEAD command for POST to establish a path with credentials before posting a body.  But in the end, any proper solution to this problem will be a significant protocol change to HTTP (since HTTP is designed for intermediaries to be able to work with unknown methods), which means we'll be living with this problem for a long time yet.
>>>>
>>>> As for NTLM, it's not going away either.  So much as people don't like it because it's not HTTP compliant auth - it's the reality we are stuck with.
>>>>
>>>> Cheers
>>>>
>>>> Adrien
>>>>
>>>>
>>>> On 30/03/2010 12:22 p.m., Jamie Lokier wrote:
>>>>
>>>>          
>>>>> Henrik Nordström wrote:
>>>>>
>>>>>
>>>>>            
>>>>>> sön 2010-03-28 klockan 15:20 +0100 skrev Jamie Lokier:
>>>>>>
>>>>>>
>>>>>>              
>>>>>>> With certain types (Microsoft) of authentication
>>>>>>>
>>>>>>>
>>>>>>>                
>>>>>> You forgot to add "which is not true HTTP authentication schemes". That
>>>>>> family of auth schemes makes many assumptions which is opposite the
>>>>>> intentions of the HTTP specifications so using them as an example when
>>>>>> trying to understand the wording of the specification text is not valid.
>>>>>>
>>>>>> Still interesting when talking about what should be said as they are a
>>>>>> reality and something HTTP has to deal with today, but not when trying
>>>>>> to understand why RFC2616 is written in a certain manner.
>>>>>>
>>>>>>
>>>>>>              
>>>>> Agreed, Microsoft's version is not standard HTTP.
>>>>>
>>>>> Regardless of why, it's important to recognise that clients have the
>>>>> option to send the whole request body if they decide to, despite one
>>>>> possible reading of that section of RFC2616 that they must abort.
>>>>>
>>>>> -- Jamie
>>>>>
>>>>>
>>>>>
>>>>>            
>>>>
>>>>          
>>>
>>>        
>>      
>    

--
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com


Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Mark Pauley
Adrien: technically you're right.  With HTTP you can't know that you need to authenticate until after you've tried.  A server could legitimately ask every nth request to authenticate, or every request with a payload of more than k bytes to authenticate, or only require authentication between 10am and 2pm or whatever.  You must try the response exactly as you want it, which is why 100-continue is so important.  It explicitly allows the server to tell the client to authenticate based only on the HTTP header.  Sorry if I'm being pedantic here, I'm probably the biggest noob on the list so I'm just spelling my understanding out explicitly.

Practically however: I've seen that Microsoft proxy servers and web servers that use NTLM authentication always ignore payload sent with the initiation of the NTLM authentication.  In essence, the first request isn't really HTTP because the client really expects the server to respond only with a 4xx message.

Of the proxy sequence:

*A: Try request
B: 407 response (list of available authentication methods)
**C: Try request (initial NTLM auth nonce / salt)
D: 407 response ( NTLM re-salt response )
*E: Try request (full NTLM challenge)
F: Success, response from endpoint

and the non-proxy sequence:
*A: Try request
B: 401 response (list of available authentication methods)
**C: Try request (initial NTLM auth nonce / salt)
D 401 response ( NTLM re-salt response )
* E: Try request (full NTLM challenge)
F: Success response.

You can see that we must send the full payload with all of the requests that are marked with a *, but we have found that we can optimize the requests that are marked with a ** by clearing the payload.  This is because according to the broken NTLM semantics, the intitial NTLM request can't result in anything but another 4xx response (and on the same TCP connection at that).  Hogwash!  We can't know what nasty state the server is in, so we should have to expect that the server may respond with a 200.. But since Microsoft is the doorkeeper here, and their own clients appear to safely follow this twisted logic, CFNetwork also saves one upload by clearing the payload.  I do not know if any pieces of the NTLM handshake interoperate with 100-continue, as I haven't tried.



On Apr 1, 2010, at 3:08 AM, Adrien de Croy wrote:

>
> keep in mind also, it's possible on a single TCP connection to be challenged for credentials multiple times.
>
> It's a common customer requirement to enable some sites for some users, and other sites for other users.  To establish who a user is requires auth.
>
> A not too uncommon scenario through a proxy follows:
>
> 1. a user connects, requests a site
> 2. site not restricted, so request granted
> 3. on same connection, user requests a restricted site which only members of group A are allowed to see
> 4. Proxy sends 407
> 5. user fills in credentials of a user in that group, request is granted
> 6. on same connection, user requests a restricted site which only members of group B are allowed to see, and the user they logged in as isn't a member
> 7. Proxy sends 407
>
> Also if a user chooses credentials that don't grant the right to view the page, a common response is another 407 - give the user another chance.  Many browsers will give up after a few tries.
>
> Proxy policy for what needs to be authed can vary on any parameter (even method, or a header or even time of day).  I had a customer yesterday who wanted to block POSTs to a specific site unless authed but only during work hours.
>
> So there's no guarantee that ANY request which is not the actual request you wish to make will elicit the same response from a proxy, or be reliably usable to pre-establish necessary credentials.
>
> Adrien
>
> On 1/04/2010 11:59 a.m., Mark Pauley wrote:
>> You're right, and we've identified these sort of issues as well.  CFNetwork (and any HTTP client in general) can't assume that they'll be given a 407, even if we know that this host has sent us one before.  Moreover, using a HEAD to 'prime' the auth also doesn't work, nor does anything similar work.  Sometimes you have actually try the full request, because HTTP allows for servers to be as pernicious as they please.
>>
>> In the end, the only time CFNetwork zeroes out the POST's length is in the case that we've already gotten a 4xx, and we are starting the NTLM auth dance, because there is no way that NTLM can just decide to not give us a 4xx once we give it the NTLM initiation in response to the initial 4xx.
>>
>>
>> On Mar 31, 2010, at 3:48 PM, Adrien de Croy wrote:
>>
>>  
>>> we saw the issue with IE + WinGate Proxy + NTLM going through to a webmail site.
>>>
>>> WinGate was configured to require auth, but credentials get inherited, and policy can be used to specify which sites need auth or not.  So not every connection was being asked to auth (407).
>>>
>>> IE was obviously working on the assumption that it would ALWAYS be challenged with a 407 on new connections.
>>>
>>> Customers complained that they couldn't reliably post messages, the webmail would randomly break.  We tracked it down to the 0 length POST on a new connection.  IE thought it would get a 407, but it didn't, and the proxy passed the 0 length post through to the webmail server, which barfed on it, with the side-effect of the browser losing the email the customer had just written.
>>>
>>> I complained about this to MS, but I think they still think it's a good idea.
>>>
>>> Our hack fix was to check UA, 0 length etc, and if we get one, arbitrarily send a 407 back.  It's really ugly and I wish to death that MS would pull this broken behaviour.
>>>
>>> Part of our selling point is that people can have policy which allows for access to some sites to be anonymous, other sites need auth etc - the admin chooses when and what needs auth.  There's no way the browser can predict it will get a 407 or not, so to send a 0-length POST is suicidal.
>>>
>>> On 1/04/2010 11:29 a.m., Mark Pauley wrote:
>>>    
>>>> CFNetwork on Mac OSX will send a zero-length post as the first leg of the 3-way auth for NTLM after it has received the initial 4xx response.
>>>>
>>>> CFNetwork ought to soon allow clients to specify Expects and handle the 100 Continue properly to reduce bandwidth on any requests with a large payload, though I can't say when 'soon' will be.  Unfortunately, the NTLM case really is dictated by Microsoft.  Their own behavior shows that they believe it's okay to send a POST with a zero length for the initial NTLM transaction, and since Microsoft is by far the largest producer of NTLM-using server software, CFNetwork just ape's IE.  What about the zero-length post breaks HTTP?  It seems like a POST that both has length 0 and Expects: 100-continue is silly, and the client ought to be able to handle either a 100-continue followed by standard HTTP response or simply just a standard HTTP response.  Garbage in, garbage out.
>>>>
>>>>
>>>> On Mar 29, 2010, at 5:42 PM, Adrien de Croy wrote:
>>>>
>>>>
>>>>      
>>>>> I did a review of what browsers do in some cases like this (e.g POST + auth/NTLM) a while back.
>>>>>
>>>>> Their behaviour varies.  For instance IE (even current) if it thinks it will get an auth challenge, will submit initial POSTs with a content-length 0 (this breaks a heap of stuff).
>>>>>
>>>>> FF and Chrome will send the whole POST body as many times as required to get through auth.
>>>>>
>>>>> Server behaviour varies also.  IIS5 for instance will send a 100 continue on all POST requests regardless of whether they contain Expects.  In fact I've never seen a browser send Expects.
>>>>>
>>>>> I wrote a I-D partly about this a while back, but it was somewhat misguided, based on my misunderstanding of the Expects mechanism.
>>>>>
>>>>> In the end, I think that's the biggest problem with Expects - people expect (pardon the pun) more from it than it delivers.  The reason the history of this list contains numerous discussions about Expects, is because of common misinterpretation.
>>>>>
>>>>> The wording in the RFC implies that Expects and 100 Continue can be used to avoid sending the body of a request.  It omits to mention that if indeed you do want to avoid this (e.g. when you get a 3xx or 4xx), you either need to disconnect and re-try, or complete the message some other way (e.g. use chunking and prematurely complete it with a 0 chunk).
>>>>>
>>>>> I would really hate to see anyone adopt the 0 chunk approach - it's just as bad as sending Content-Length: 0 when the browser thinks the proxy or server will send an auth challenge.  Browsers second-guessing proxies / servers is not a good option, and when the browser gets it wrong, things get ugly.  We had to write a hack into WinGate to cope with the IE misbehaviour (WinGate only sends an auth challenge when policy dictates auth is required - the browser cannot possibly predict this, and a 0 length post is valid, but breaks websites).  Sending a 0 length chunk tells the proxy the message is complete.  It could then go through many stages of processing that is really undesirable.  There's no way to signal an abort and maintain a connection.
>>>>>
>>>>> In the end, the best option IMO is to send the whole thing each time (as per FF and Chrome - sorry don't know what Opera does).  It can be hideously painful with chained connection-oriented auth over a slow link.
>>>>>
>>>>> So actually there's no clean solution to this problem.  I proposed a while back a kind of HEAD command for POST to establish a path with credentials before posting a body.  But in the end, any proper solution to this problem will be a significant protocol change to HTTP (since HTTP is designed for intermediaries to be able to work with unknown methods), which means we'll be living with this problem for a long time yet.
>>>>>
>>>>> As for NTLM, it's not going away either.  So much as people don't like it because it's not HTTP compliant auth - it's the reality we are stuck with.
>>>>>
>>>>> Cheers
>>>>>
>>>>> Adrien
>>>>>
>>>>>
>>>>> On 30/03/2010 12:22 p.m., Jamie Lokier wrote:
>>>>>
>>>>>        
>>>>>> Henrik Nordström wrote:
>>>>>>
>>>>>>
>>>>>>          
>>>>>>> sön 2010-03-28 klockan 15:20 +0100 skrev Jamie Lokier:
>>>>>>>
>>>>>>>
>>>>>>>            
>>>>>>>> With certain types (Microsoft) of authentication
>>>>>>>>
>>>>>>>>
>>>>>>>>              
>>>>>>> You forgot to add "which is not true HTTP authentication schemes". That
>>>>>>> family of auth schemes makes many assumptions which is opposite the
>>>>>>> intentions of the HTTP specifications so using them as an example when
>>>>>>> trying to understand the wording of the specification text is not valid.
>>>>>>>
>>>>>>> Still interesting when talking about what should be said as they are a
>>>>>>> reality and something HTTP has to deal with today, but not when trying
>>>>>>> to understand why RFC2616 is written in a certain manner.
>>>>>>>
>>>>>>>
>>>>>>>            
>>>>>> Agreed, Microsoft's version is not standard HTTP.
>>>>>>
>>>>>> Regardless of why, it's important to recognise that clients have the
>>>>>> option to send the whole request body if they decide to, despite one
>>>>>> possible reading of that section of RFC2616 that they must abort.
>>>>>>
>>>>>> -- Jamie
>>>>>>
>>>>>>
>>>>>>
>>>>>>          
>>>>>
>>>>>        
>>>>
>>>>      
>>>    
>>  
>
> --
> Adrien de Croy - WinGate Proxy Server - http://www.wingate.com
>
>


Reply | Threaded
Open this post in threaded view
|

Re: 100 Continue and Expects

Jamie Lokier
Mark Pauley wrote:
> Practically however: I've seen that Microsoft proxy servers and web
> servers that use NTLM authentication always ignore payload sent with
> the initiation of the NTLM authentication.  In essence, the first
> request isn't really HTTP because the client really expects the
> server to respond only with a 4xx message.

A proxy is free to forward your request to IIS between 10am and 2pm,
and to forward your request to Apache on a Linux box with no
authentication after 2pm.  So it is, alas, broken in this scenario.
But that's the nature of the NTLM beast.

-- Jamie

12