HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Bob Briscoe
HTTP/2 folks,

As I said, consider this as a late review from a clueful but fresh pair of eyes.
My main concerns with the draft are:
* extensibility (previous posting)
* flow control (this posting - apologies for the length - I've tried to explain properly)
* numerous open issues left dangling (see subsequent postings)

===HTTP/2 FLOW CONTROL===

The term 'window' as used throughout is incorrect and highly confusing, in:
* 'flow control window' (44 occurrences),
* 'initial window size' (5),
* or just 'window size' (8)
* and the terms WINDOW_UPDATE and SETTINGS_INITIAL_WINDOW.

The HTTP/2 WINDOW_UPDATE mechanism constrains HTTP/2 to use only credit-based flow control, not window-based. At one point, it actually says it is credit-based (in flow control principle #2 < https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-5.2.1 > ), but otherwise it incorrectly uses the term window.

This is not just an issue of terminology. The more I re-read the flow control sections the more I became convinced that this terminology is not just /confusing/, rather it's evidence of /confusion/. It raises the questions
* "Is HTTP/2 capable of the flow control it says it's capable of?"
* "What type of flow-control protocol ought HTTP/2 to be capable of?"
* "Can the WINDOW_UPDATE frame support the flow-control that HTTP/2 needs?"

To address these questions, it may help if I separate the two different cases HTTP/2 flow control attempts to cover (my own separation, not from the draft):

a) Intermediate buffer control
Here, a stream's flow enters /and/ leaves a buffer (e.g. at the app-layer of an intermediate node).

b) Flow control by the ultimate client app.
Here flow never releases memory (at least not during the life of the connection). The flow is solely consuming more and more memory (e.g. data being rendered into a client app's memory).

==a) Intermediate buffer control==
For this, sliding window-based flow control would be appropriate, because the goal is to keep the e2e pipeline full without wasting buffer.

Let me prove HTTP/2 cannot do window flow control. For window flow control, the sender needs to be able to advance both the leading and trailing edges of the window. In the draft:
* WINDOW_UPDATE frames can only advance the leading edge of a 'window' (and they are constrained to positive values).
* To advance the trailing edge, window flow control would need a continuous stream of acknowledgements back to the sender (like TCP). The draft does not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at the transport layer, so the sending app-layer cannot advance the trailing edge of a 'window'.

So the protocol can only support credit-based flow control. It is incapable of supporting window flow control.

Next, I don't understand how a receiver can set the credit in 'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to answer the question "How much more can I send than I have seen ACK'd?" that would be easy. But because the protocol is restricted to credit, the sender needs the receiver to answer the much harder open-ended question, "How much more can I send?" So the sender needs the receiver to know how many ACKs the sender has seen, but neither of them know that.

The receiver can try, by taking a guess at the bandwidth-delay product, and adjusting the guess up or down, depending on whether its buffer is growing or shrinking. But this only works if the unknown bandwidth-delay product stays constant.

However, BDP will usually be highly variable, as other streams come and go. So, in the time it takes to get a good estimate of the per-stream BDP, it will probably have changed radically, or the stream will most likely have finished anyway. This is why TCP bases flow control on a window, not credit. By complementing window updates with ACK stream info, a TCP sender has sufficient info to control the flow.

The draft is indeed correct when it says:
"   this can lead to suboptimal use of available
   network resources if flow control is enabled without knowledge of the
   bandwidth-delay product (see [RFC7323]).
"
Was this meant to be a veiled criticism of the protocol's own design? A credit-based flow control protocol like that in the draft does not provide sufficient information for either end to estimate the bandwidth-delay product, given it will be varying rapidly.

==b) Control by the ultimate client app==
For this case, I believe neither window nor credit-based flow control is appropriate:
* There is no memory management issue at the client end - even if there's a separate HTTP/2 layer of memory between TCP and the app, it would be pointless to limit the memory used by HTTP/2, because the data is still going to sit in the same user-space memory (or at least about the same amount of memory) when HTTP/2 passes it over for rendering.
* Nonetheless, the receiving client does need to send messages to the sender to supplement stream priorities, by notifying when the state of the receiving application has changed (e.g. if the user's focus switches from one browser tab to another).
* However, credit-based flow control would be very sluggish for such control, because credit cannot be taken back once it has been given (except HTTP/2 allows SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure that hits all streams together).

==Flow control problem summary==

With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.

==Towards a solution?==

I think 'type-a' flow control (for intermediate buffer control) does not need to be at stream-granularity. Indeed, I suspect a proxy could control its app-layer buffering by controlling the receive window of the incoming TCP connection. Has anyone assessed whether this would be sufficient?

I can understand the need for 'type-b' per-stream flow control (by the ultimate client endpoint). Perhaps it would be useful for the receiver to emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream PRIORITY would be sufficient? Either would minimise the response time to a half round trip. Whereas credit flow-control will be much more sluggish (see 'Flow control problem summary').

Either approach would correctly propagate e2e. An intermediate node would naturally tend to prioritise incoming streams that fed into prioritised outgoing streams, so priority updates would tend to propagate from the ultimate receiver, through intermediate nodes, up to the ultimate sender.

==Flow control coverage==

The draft exempts all TCP payload bytes from flow control except HTTP/2 data frames. No rationale is given for this decision. The draft says it's important to manage per-stream memory, then it exempts all the frame types except data, even tho each byte of a non-data frame consumes no less memory than a byte of a data frame.

What message does this put out? "Flow control is not important for one type of bytes with unlimited total size, but flow control is so important that it has to be mandatory for the other type of bytes."

It is certainly critical that WINDOW_UPDATE messages are not covered by flow control, otherwise there would be a real risk of deadlock. It might be that there are dependencies on other frame types that would lead to a dependency loop and deadlock. It would be good to know what the rationale behind these rules was.

==Theory?==

I am concerned that HTTP/2 flow control may have entered new theoretical territory, without suitable proof of safety. The only reassurance we have is one implementation of a flow control algorithm (SPDY), and the anecdotal non-evidence that no-one using SPDY has noticed a deadlock yet (however, is anyone monitoring for deadlocks?).

Whereas SPDY has been an existence proof that an approach like http/2 'works', so far all the flow control algos have been pretty much identical (I think that's true?). I am concerned that the draft takes the InterWeb into uncharted waters, because it allows unconstrained diversity in flow control algos, which is an untested degree of freedom.

The only constraints the draft sets are:
* per-stream flow control is mandatory
* the only protocol message for flow control algos to use is the WINDOW_UPDATE credit message, which cannot be negative
* no constraints on flow control algorithms.
* and all this must work within the outer flow control constraints of TCP.

Some algos might use priority messages to make flow control assumptions. While other algos might associate PRI and WINDOW_UPDATE with different meanings. What confidence do we have that everyone's optimisation algorithms will interoperate? Do we know there will not be certain types of application where deadlock is likely?

"   When using flow
   control, the receiver MUST read from the TCP receive buffer in a
   timely fashion.  Failure to do so could lead to a deadlock when
   critical frames, such as WINDOW_UPDATE, are not read and acted upon.
"
I've been convinced (offlist) that deadlock will not occur as long as the app consumes data 'greedily' from TCP. That has since been articulated in the above normative text. But how sure can we be that every implementer's different interpretations of 'timely' will still prevent deadlock?

Until a good autotuning algorithm for TCP receive window management was developed, good window management code was nearly non-existent. Managing hundreds of interdependent stream buffers is a much harder problem. But implementers are being allowed to just 'Go forth and innovate'. This might work if everyone copies available open source algo(s). But they might not, and they don't have to.

This all seems like 'flying by the seat of the pants'.

==Mandatory Flow Control? ==

"      3. [...] A sender
       MUST respect flow control limits imposed by a receiver."
This ought to be a 'SHOULD' because it is contradicted later - if settings change.

"   6.  Flow control cannot be disabled."
Also effectively contradicted half a page later:
"   Deployments that do not require this capability can advertise a flow
   control window of the maximum size (2^31-1), and by maintaining this
   window by sending a WINDOW_UPDATE frame when any data is received.
   This effectively disables flow control for that receiver."

And contradicted in the definition of half closed (remote):
"  half closed (remote):
      [...] an endpoint is no longer
      obligated to maintain a receiver flow control window.
"

And contradicted in 8.3. The CONNECT Method, which says:
"  Frame types other than DATA
   or stream management frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY)
   MUST NOT be sent on a connected stream, and MUST be treated as a
   stream error (Section 5.4.2) if received.
"

Why is flow control so important that it's mandatory, but so unimportant that you MUST NOT do it when using TLS e2e?

Going back to the earlier quote about using the max window size, it seems perverse for the spec to require endpoints to go through the motions of flow control, even if they arrange for it to affect nothing, but to still require implementation complexity and bandwidth waste with a load of redundant WINDOW_UPDATE frames.

HTTP is used on a wide range of devices, down to the very small and challenged. HTTP/2 might be desirable in such cases, because of the improved efficiency (e.g. header compression), but in many cases the stream model may not be complex enough to need stream flow control.

So why not make flow control optional on the receiving side, but mandatory to implement on the sending side? Then an implementation could have no machinery for tuning window sizes, but it would respond correctly to those set by the other end, which requires much simpler code.

If a receiving implemention chose not to do stream flow control, it could still control flow at the connection (stream 0) level, or at least at the TCP level.


==Inefficiency?==

5.2. Flow Control
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.

==Flow Control Requirements===

I'm not convinced that clear understanding of flow control requirements has driven flow control design decisions.

The draft states various needs for flow-control without giving me a feel of confidence that it has separated out the different cases, and chosen a protocol suitable for each. I tried to go back to the early draft on flow control requirements < http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01 >, and I was not impressed.

I have quoted below the various sentences in the draft that state what flow control is believed to be for. Below that, I have attempted to crystalize out the different concepts, each of which I have tagged within the quotes.

* 2. HTTP/2 Protocol Overview says
  "Flow control and prioritization ensure that it is possible to efficiently use multiplexed streams. [Y]
   Flow control (Section 5.2) helps to ensure that only data that can be used by a receiver is transmitted. [X]"
* 5.2. Flow Control says:
  "Using streams for multiplexing introduces contention over use of the TCP connection [X], resulting in blocked streams [Z]. A flow control scheme ensures that streams on the same connection do not destructively interfere with each other [Z]."
* 5.2.2. Appropriate Use of Flow Control
"  Flow control is defined to protect endpoints that are operating under
   resource constraints.  For example, a proxy needs to share memory
   between many connections, and also might have a slow upstream
   connection and a fast downstream one [Y].  Flow control addresses cases
   where the receiver is unable to process data on one stream, yet wants
   to continue to process other streams in the same connection [X]."
"  Deployments with constrained resources (for example, memory) can
   employ flow control to limit the amount of memory a peer can consume. [Y]

Each requirement has been tagged as follows:
[X] Notification of the receiver's changing utility for each stream
[Y] Prioritisation of streams due to contention over the streaming capacity available to the whole connection.
[Z] Ensuring one stream is not blocked by another.

[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y] sounds more like optimisation across a continuous spectrum.


Regards



Bob

________________________________________________________________
Bob Briscoe,                                                  BT

Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Bob Briscoe
HTTP/2 folks,

I know extensibility had already been discussed and put to bed, so the WG is entitled to rule out opening healed wounds.

But have points like those I've made about flow control been raised before? Please argue. I may be wrong. Discussion can go on in parallel to the RFC publication process, even tho the process doesn't /require/ you to talk to me.

If I'm right, then implementers are being mandated to write complex flow control code, when it might have little bearing on the performance benefits measured for http/2.

Even if I'm right, and the WG goes ahead anyway, /I/ will understand. My review came in after your deadline.

However, bear in mind that the Webosphere might not be so forgiving. If h2 goes ahead when potential problems have been identified, it could get a bad reputation simply due to the uncertainty, just when you want more people to take it up and try it out. Given you've put in a few person-years of effort, I would have thought you would not want to risk a reputation flop.

I'm trying to help - I just can't go any faster.


Bob

At 14:43 06/03/2015, Bob Briscoe wrote:
HTTP/2 folks,

As I said, consider this as a late review from a clueful but fresh pair of eyes.
My main concerns with the draft are:
* extensibility (previous posting)
* flow control (this posting - apologies for the length - I've tried to explain properly)
* numerous open issues left dangling (see subsequent postings)

===HTTP/2 FLOW CONTROL===

The term 'window' as used throughout is incorrect and highly confusing, in:
* 'flow control window' (44 occurrences),
* 'initial window size' (5),
* or just 'window size' (8)
* and the terms WINDOW_UPDATE and SETTINGS_INITIAL_WINDOW.

The HTTP/2 WINDOW_UPDATE mechanism constrains HTTP/2 to use only credit-based flow control, not window-based. At one point, it actually says it is credit-based (in flow control principle #2 < https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-5.2.1 > ), but otherwise it incorrectly uses the term window.

This is not just an issue of terminology. The more I re-read the flow control sections the more I became convinced that this terminology is not just /confusing/, rather it's evidence of /confusion/. It raises the questions
* "Is HTTP/2 capable of the flow control it says it's capable of?"
* "What type of flow-control protocol ought HTTP/2 to be capable of?"
* "Can the WINDOW_UPDATE frame support the flow-control that HTTP/2 needs?"

To address these questions, it may help if I separate the two different cases HTTP/2 flow control attempts to cover (my own separation, not from the draft):

a) Intermediate buffer control
Here, a stream's flow enters /and/ leaves a buffer (e.g. at the app-layer of an intermediate node).

b) Flow control by the ultimate client app.
Here flow never releases memory (at least not during the life of the connection). The flow is solely consuming more and more memory (e.g. data being rendered into a client app's memory).

==a) Intermediate buffer control==
For this, sliding window-based flow control would be appropriate, because the goal is to keep the e2e pipeline full without wasting buffer.

Let me prove HTTP/2 cannot do window flow control. For window flow control, the sender needs to be able to advance both the leading and trailing edges of the window. In the draft:
* WINDOW_UPDATE frames can only advance the leading edge of a 'window' (and they are constrained to positive values).
* To advance the trailing edge, window flow control would need a continuous stream of acknowledgements back to the sender (like TCP). The draft does not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at the transport layer, so the sending app-layer cannot advance the trailing edge of a 'window'.

So the protocol can only support credit-based flow control. It is incapable of supporting window flow control.

Next, I don't understand how a receiver can set the credit in 'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to answer the question "How much more can I send than I have seen ACK'd?" that would be easy. But because the protocol is restricted to credit, the sender needs the receiver to answer the much harder open-ended question, "How much more can I send?" So the sender needs the receiver to know how many ACKs the sender has seen, but neither of them know that.

The receiver can try, by taking a guess at the bandwidth-delay product, and adjusting the guess up or down, depending on whether its buffer is growing or shrinking. But this only works if the unknown bandwidth-delay product stays constant.

However, BDP will usually be highly variable, as other streams come and go. So, in the time it takes to get a good estimate of the per-stream BDP, it will probably have changed radically, or the stream will most likely have finished anyway. This is why TCP bases flow control on a window, not credit. By complementing window updates with ACK stream info, a TCP sender has sufficient info to control the flow.

The draft is indeed correct when it says:
"   this can lead to suboptimal use of available
   network resources if flow control is enabled without knowledge of the
   bandwidth-delay product (see [RFC7323]).
"
Was this meant to be a veiled criticism of the protocol's own design? A credit-based flow control protocol like that in the draft does not provide sufficient information for either end to estimate the bandwidth-delay product, given it will be varying rapidly.

==b) Control by the ultimate client app==
For this case, I believe neither window nor credit-based flow control is appropriate:
* There is no memory management issue at the client end - even if there's a separate HTTP/2 layer of memory between TCP and the app, it would be pointless to limit the memory used by HTTP/2, because the data is still going to sit in the same user-space memory (or at least about the same amount of memory) when HTTP/2 passes it over for rendering.
* Nonetheless, the receiving client does need to send messages to the sender to supplement stream priorities, by notifying when the state of the receiving application has changed (e.g. if the user's focus switches from one browser tab to another).
* However, credit-based flow control would be very sluggish for such control, because credit cannot be taken back once it has been given (except HTTP/2 allows SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure that hits all streams together).

==Flow control problem summary==

With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.

==Towards a solution?==

I think 'type-a' flow control (for intermediate buffer control) does not need to be at stream-granularity. Indeed, I suspect a proxy could control its app-layer buffering by controlling the receive window of the incoming TCP connection. Has anyone assessed whether this would be sufficient?

I can understand the need for 'type-b' per-stream flow control (by the ultimate client endpoint). Perhaps it would be useful for the receiver to emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream PRIORITY would be sufficient? Either would minimise the response time to a half round trip. Whereas credit flow-control will be much more sluggish (see 'Flow control problem summary').

Either approach would correctly propagate e2e. An intermediate node would naturally tend to prioritise incoming streams that fed into prioritised outgoing streams, so priority updates would tend to propagate from the ultimate receiver, through intermediate nodes, up to the ultimate sender.

==Flow control coverage==

The draft exempts all TCP payload bytes from flow control except HTTP/2 data frames. No rationale is given for this decision. The draft says it's important to manage per-stream memory, then it exempts all the frame types except data, even tho each byte of a non-data frame consumes no less memory than a byte of a data frame.

What message does this put out? "Flow control is not important for one type of bytes with unlimited total size, but flow control is so important that it has to be mandatory for the other type of bytes."

It is certainly critical that WINDOW_UPDATE messages are not covered by flow control, otherwise there would be a real risk of deadlock. It might be that there are dependencies on other frame types that would lead to a dependency loop and deadlock. It would be good to know what the rationale behind these rules was.

==Theory?==

I am concerned that HTTP/2 flow control may have entered new theoretical territory, without suitable proof of safety. The only reassurance we have is one implementation of a flow control algorithm (SPDY), and the anecdotal non-evidence that no-one using SPDY has noticed a deadlock yet (however, is anyone monitoring for deadlocks?).

Whereas SPDY has been an existence proof that an approach like http/2 'works', so far all the flow control algos have been pretty much identical (I think that's true?). I am concerned that the draft takes the InterWeb into uncharted waters, because it allows unconstrained diversity in flow control algos, which is an untested degree of freedom.

The only constraints the draft sets are:
* per-stream flow control is mandatory
* the only protocol message for flow control algos to use is the WINDOW_UPDATE credit message, which cannot be negative
* no constraints on flow control algorithms.
* and all this must work within the outer flow control constraints of TCP.

Some algos might use priority messages to make flow control assumptions. While other algos might associate PRI and WINDOW_UPDATE with different meanings. What confidence do we have that everyone's optimisation algorithms will interoperate? Do we know there will not be certain types of application where deadlock is likely?

"   When using flow
   control, the receiver MUST read from the TCP receive buffer in a
   timely fashion.  Failure to do so could lead to a deadlock when
   critical frames, such as WINDOW_UPDATE, are not read and acted upon.
"
I've been convinced (offlist) that deadlock will not occur as long as the app consumes data 'greedily' from TCP. That has since been articulated in the above normative text. But how sure can we be that every implementer's different interpretations of 'timely' will still prevent deadlock?

Until a good autotuning algorithm for TCP receive window management was developed, good window management code was nearly non-existent. Managing hundreds of interdependent stream buffers is a much harder problem. But implementers are being allowed to just 'Go forth and innovate'. This might work if everyone copies available open source algo(s). But they might not, and they don't have to.

This all seems like 'flying by the seat of the pants'.

==Mandatory Flow Control? ==

"      3. [...] A sender
       MUST respect flow control limits imposed by a receiver."
This ought to be a 'SHOULD' because it is contradicted later - if settings change.

"   6.  Flow control cannot be disabled."
Also effectively contradicted half a page later:
"   Deployments that do not require this capability can advertise a flow
   control window of the maximum size (2^31-1), and by maintaining this
   window by sending a WINDOW_UPDATE frame when any data is received.
   This effectively disables flow control for that receiver."

And contradicted in the definition of half closed (remote):
"  half closed (remote):
      [...] an endpoint is no longer
      obligated to maintain a receiver flow control window.
"

And contradicted in 8.3. The CONNECT Method, which says:
"  Frame types other than DATA
   or stream management frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY)
   MUST NOT be sent on a connected stream, and MUST be treated as a
   stream error (Section 5.4.2) if received.
"

Why is flow control so important that it's mandatory, but so unimportant that you MUST NOT do it when using TLS e2e?

Going back to the earlier quote about using the max window size, it seems perverse for the spec to require endpoints to go through the motions of flow control, even if they arrange for it to affect nothing, but to still require implementation complexity and bandwidth waste with a load of redundant WINDOW_UPDATE frames.

HTTP is used on a wide range of devices, down to the very small and challenged. HTTP/2 might be desirable in such cases, because of the improved efficiency (e.g. header compression), but in many cases the stream model may not be complex enough to need stream flow control.

So why not make flow control optional on the receiving side, but mandatory to implement on the sending side? Then an implementation could have no machinery for tuning window sizes, but it would respond correctly to those set by the other end, which requires much simpler code.

If a receiving implemention chose not to do stream flow control, it could still control flow at the connection (stream 0) level, or at least at the TCP level.


==Inefficiency?==

5.2. Flow Control
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.

==Flow Control Requirements===

I'm not convinced that clear understanding of flow control requirements has driven flow control design decisions.

The draft states various needs for flow-control without giving me a feel of confidence that it has separated out the different cases, and chosen a protocol suitable for each. I tried to go back to the early draft on flow control requirements < http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01 >, and I was not impressed.

I have quoted below the various sentences in the draft that state what flow control is believed to be for. Below that, I have attempted to crystalize out the different concepts, each of which I have tagged within the quotes.

* 2. HTTP/2 Protocol Overview says
  "Flow control and prioritization ensure that it is possible to efficiently use multiplexed streams. [Y]
   Flow control (Section 5.2) helps to ensure that only data that can be used by a receiver is transmitted. [X]"
* 5.2. Flow Control says:
  "Using streams for multiplexing introduces contention over use of the TCP connection [X], resulting in blocked streams [Z]. A flow control scheme ensures that streams on the same connection do not destructively interfere with each other [Z]."
* 5.2.2. Appropriate Use of Flow Control
"  Flow control is defined to protect endpoints that are operating under
   resource constraints.  For example, a proxy needs to share memory
   between many connections, and also might have a slow upstream
   connection and a fast downstream one [Y].  Flow control addresses cases
   where the receiver is unable to process data on one stream, yet wants
   to continue to process other streams in the same connection [X]."
"  Deployments with constrained resources (for example, memory) can
   employ flow control to limit the amount of memory a peer can consume. [Y]

Each requirement has been tagged as follows:
[X] Notification of the receiver's changing utility for each stream
[Y] Prioritisation of streams due to contention over the streaming capacity available to the whole connection.
[Z] Ensuring one stream is not blocked by another.

[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y] sounds more like optimisation across a continuous spectrum.


Regards



Bob

________________________________________________________________
Bob Briscoe,                                                  BT

________________________________________________________________
Bob Briscoe,                                                  BT

Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Patrick McManus-3
Hi Bob - I think your comments are appreciated. Its just one of those things where people have dispersed to other things and aren't necessarily in a place to revisit all the ground work again at this stage for a new go round. It was in large part the operational feedback and needs of the google.com team, who has a lot of experience operating spdy at scale, that created the flow control provisions. hopefully those folks will chime in more authoritatively than my musings below:

I'm sure there is quite a bit to learn here - indeed poorly configured use of the window_update mechanism has been (predictably) a source of unintended bottlenecks during both spdy and h2 trials. The spec does try and highlight that there can be dragons here and implementations that don't need the features it can bring should provide essentially infinite credits to steer clear of them.

During the various trials I've seen h2 per stream flow control deployed successfully for a couple of use cases - both of them essentially deal with unsolicited data.

The primary one is essentially a more flexible version of h1's 100-continue. When a client presents a large message body (e.g. a file upload) a multiplexing server needs a way of saying "these are how many buffers I've got available while I figure out where I'm going to sink this incoming data (perhaps to another server I need to connect to)". Presenting this on a per-stream basis allows the server to limit one stream while another (with a distinct sink) can proceed independently.  IMO this value should represent resources available and should be independent of BDP. This is why in practice you see clients with extremely large stream windows - most circumstances just want the data to flow at line rate (as you describe) and aren't trying to invoke flow control. The firefox default window is 256MB per stream - that's not going to slow down the sender nor require frequent window_update generation.

The other use case is when the server pushes resources at a client without them being requested, which is a new feature of h2. This is conceptually similar to the server receiving a POST - suddenly there is a large amount of inbound data that the implementation might not have the resources to store completely. We can't just let TCP flow control take care of the situation because the TCP session is being multiplexed between multiple streams that need to be serviced. In this case the client accepts "some" of the stream based on policy and resource availability and can leave the stream in limbo until an event comes along that tells it to resume the transfer by issuing credits or reject it via cancel.

hth a least a bit.


On Tue, Mar 10, 2015 at 4:31 PM, Bob Briscoe <[hidden email]> wrote:
HTTP/2 folks,

I know extensibility had already been discussed and put to bed, so the WG is entitled to rule out opening healed wounds.

But have points like those I've made about flow control been raised before? Please argue. I may be wrong. Discussion can go on in parallel to the RFC publication process, even tho the process doesn't /require/ you to talk to me.

If I'm right, then implementers are being mandated to write complex flow control code, when it might have little bearing on the performance benefits measured for http/2.

Even if I'm right, and the WG goes ahead anyway, /I/ will understand. My review came in after your deadline.

However, bear in mind that the Webosphere might not be so forgiving. If h2 goes ahead when potential problems have been identified, it could get a bad reputation simply due to the uncertainty, just when you want more people to take it up and try it out. Given you've put in a few person-years of effort, I would have thought you would not want to risk a reputation flop.

I'm trying to help - I just can't go any faster.


Bob


At 14:43 06/03/2015, Bob Briscoe wrote:
HTTP/2 folks,

As I said, consider this as a late review from a clueful but fresh pair of eyes.
My main concerns with the draft are:
* extensibility (previous posting)
* flow control (this posting - apologies for the length - I've tried to explain properly)
* numerous open issues left dangling (see subsequent postings)

===HTTP/2 FLOW CONTROL===

The term 'window' as used throughout is incorrect and highly confusing, in:
* 'flow control window' (44 occurrences),
* 'initial window size' (5),
* or just 'window size' (8)
* and the terms WINDOW_UPDATE and SETTINGS_INITIAL_WINDOW.

The HTTP/2 WINDOW_UPDATE mechanism constrains HTTP/2 to use only credit-based flow control, not window-based. At one point, it actually says it is credit-based (in flow control principle #2 < https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-5.2.1 > ), but otherwise it incorrectly uses the term window.

This is not just an issue of terminology. The more I re-read the flow control sections the more I became convinced that this terminology is not just /confusing/, rather it's evidence of /confusion/. It raises the questions
* "Is HTTP/2 capable of the flow control it says it's capable of?"
* "What type of flow-control protocol ought HTTP/2 to be capable of?"
* "Can the WINDOW_UPDATE frame support the flow-control that HTTP/2 needs?"

To address these questions, it may help if I separate the two different cases HTTP/2 flow control attempts to cover (my own separation, not from the draft):

a) Intermediate buffer control
Here, a stream's flow enters /and/ leaves a buffer (e.g. at the app-layer of an intermediate node).

b) Flow control by the ultimate client app.
Here flow never releases memory (at least not during the life of the connection). The flow is solely consuming more and more memory (e.g. data being rendered into a client app's memory).

==a) Intermediate buffer control==
For this, sliding window-based flow control would be appropriate, because the goal is to keep the e2e pipeline full without wasting buffer.

Let me prove HTTP/2 cannot do window flow control. For window flow control, the sender needs to be able to advance both the leading and trailing edges of the window. In the draft:
* WINDOW_UPDATE frames can only advance the leading edge of a 'window' (and they are constrained to positive values).
* To advance the trailing edge, window flow control would need a continuous stream of acknowledgements back to the sender (like TCP). The draft does not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at the transport layer, so the sending app-layer cannot advance the trailing edge of a 'window'.

So the protocol can only support credit-based flow control. It is incapable of supporting window flow control.

Next, I don't understand how a receiver can set the credit in 'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to answer the question "How much more can I send than I have seen ACK'd?" that would be easy. But because the protocol is restricted to credit, the sender needs the receiver to answer the much harder open-ended question, "How much more can I send?" So the sender needs the receiver to know how many ACKs the sender has seen, but neither of them know that.

The receiver can try, by taking a guess at the bandwidth-delay product, and adjusting the guess up or down, depending on whether its buffer is growing or shrinking. But this only works if the unknown bandwidth-delay product stays constant.

However, BDP will usually be highly variable, as other streams come and go. So, in the time it takes to get a good estimate of the per-stream BDP, it will probably have changed radically, or the stream will most likely have finished anyway. This is why TCP bases flow control on a window, not credit. By complementing window updates with ACK stream info, a TCP sender has sufficient info to control the flow.

The draft is indeed correct when it says:
"   this can lead to suboptimal use of available
   network resources if flow control is enabled without knowledge of the
   bandwidth-delay product (see [RFC7323]).
"
Was this meant to be a veiled criticism of the protocol's own design? A credit-based flow control protocol like that in the draft does not provide sufficient information for either end to estimate the bandwidth-delay product, given it will be varying rapidly.

==b) Control by the ultimate client app==
For this case, I believe neither window nor credit-based flow control is appropriate:
* There is no memory management issue at the client end - even if there's a separate HTTP/2 layer of memory between TCP and the app, it would be pointless to limit the memory used by HTTP/2, because the data is still going to sit in the same user-space memory (or at least about the same amount of memory) when HTTP/2 passes it over for rendering.
* Nonetheless, the receiving client does need to send messages to the sender to supplement stream priorities, by notifying when the state of the receiving application has changed (e.g. if the user's focus switches from one browser tab to another).
* However, credit-based flow control would be very sluggish for such control, because credit cannot be taken back once it has been given (except HTTP/2 allows SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure that hits all streams together).

==Flow control problem summary==

With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.

==Towards a solution?==

I think 'type-a' flow control (for intermediate buffer control) does not need to be at stream-granularity. Indeed, I suspect a proxy could control its app-layer buffering by controlling the receive window of the incoming TCP connection. Has anyone assessed whether this would be sufficient?

I can understand the need for 'type-b' per-stream flow control (by the ultimate client endpoint). Perhaps it would be useful for the receiver to emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream PRIORITY would be sufficient? Either would minimise the response time to a half round trip. Whereas credit flow-control will be much more sluggish (see 'Flow control problem summary').

Either approach would correctly propagate e2e. An intermediate node would naturally tend to prioritise incoming streams that fed into prioritised outgoing streams, so priority updates would tend to propagate from the ultimate receiver, through intermediate nodes, up to the ultimate sender.

==Flow control coverage==

The draft exempts all TCP payload bytes from flow control except HTTP/2 data frames. No rationale is given for this decision. The draft says it's important to manage per-stream memory, then it exempts all the frame types except data, even tho each byte of a non-data frame consumes no less memory than a byte of a data frame.

What message does this put out? "Flow control is not important for one type of bytes with unlimited total size, but flow control is so important that it has to be mandatory for the other type of bytes."

It is certainly critical that WINDOW_UPDATE messages are not covered by flow control, otherwise there would be a real risk of deadlock. It might be that there are dependencies on other frame types that would lead to a dependency loop and deadlock. It would be good to know what the rationale behind these rules was.

==Theory?==

I am concerned that HTTP/2 flow control may have entered new theoretical territory, without suitable proof of safety. The only reassurance we have is one implementation of a flow control algorithm (SPDY), and the anecdotal non-evidence that no-one using SPDY has noticed a deadlock yet (however, is anyone monitoring for deadlocks?).

Whereas SPDY has been an existence proof that an approach like http/2 'works', so far all the flow control algos have been pretty much identical (I think that's true?). I am concerned that the draft takes the InterWeb into uncharted waters, because it allows unconstrained diversity in flow control algos, which is an untested degree of freedom.

The only constraints the draft sets are:
* per-stream flow control is mandatory
* the only protocol message for flow control algos to use is the WINDOW_UPDATE credit message, which cannot be negative
* no constraints on flow control algorithms.
* and all this must work within the outer flow control constraints of TCP.

Some algos might use priority messages to make flow control assumptions. While other algos might associate PRI and WINDOW_UPDATE with different meanings. What confidence do we have that everyone's optimisation algorithms will interoperate? Do we know there will not be certain types of application where deadlock is likely?

"   When using flow
   control, the receiver MUST read from the TCP receive buffer in a
   timely fashion.  Failure to do so could lead to a deadlock when
   critical frames, such as WINDOW_UPDATE, are not read and acted upon.
"
I've been convinced (offlist) that deadlock will not occur as long as the app consumes data 'greedily' from TCP. That has since been articulated in the above normative text. But how sure can we be that every implementer's different interpretations of 'timely' will still prevent deadlock?

Until a good autotuning algorithm for TCP receive window management was developed, good window management code was nearly non-existent. Managing hundreds of interdependent stream buffers is a much harder problem. But implementers are being allowed to just 'Go forth and innovate'. This might work if everyone copies available open source algo(s). But they might not, and they don't have to.

This all seems like 'flying by the seat of the pants'.

==Mandatory Flow Control? ==

"      3. [...] A sender
       MUST respect flow control limits imposed by a receiver."
This ought to be a 'SHOULD' because it is contradicted later - if settings change.

"   6.  Flow control cannot be disabled."
Also effectively contradicted half a page later:
"   Deployments that do not require this capability can advertise a flow
   control window of the maximum size (2^31-1), and by maintaining this
   window by sending a WINDOW_UPDATE frame when any data is received.
   This effectively disables flow control for that receiver."

And contradicted in the definition of half closed (remote):
"  half closed (remote):
      [...] an endpoint is no longer
      obligated to maintain a receiver flow control window.
"

And contradicted in 8.3. The CONNECT Method, which says:
"  Frame types other than DATA
   or stream management frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY)
   MUST NOT be sent on a connected stream, and MUST be treated as a
   stream error (Section 5.4.2) if received.
"

Why is flow control so important that it's mandatory, but so unimportant that you MUST NOT do it when using TLS e2e?

Going back to the earlier quote about using the max window size, it seems perverse for the spec to require endpoints to go through the motions of flow control, even if they arrange for it to affect nothing, but to still require implementation complexity and bandwidth waste with a load of redundant WINDOW_UPDATE frames.

HTTP is used on a wide range of devices, down to the very small and challenged. HTTP/2 might be desirable in such cases, because of the improved efficiency (e.g. header compression), but in many cases the stream model may not be complex enough to need stream flow control.

So why not make flow control optional on the receiving side, but mandatory to implement on the sending side? Then an implementation could have no machinery for tuning window sizes, but it would respond correctly to those set by the other end, which requires much simpler code.

If a receiving implemention chose not to do stream flow control, it could still control flow at the connection (stream 0) level, or at least at the TCP level.


==Inefficiency?==

5.2. Flow Control
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.

==Flow Control Requirements===

I'm not convinced that clear understanding of flow control requirements has driven flow control design decisions.

The draft states various needs for flow-control without giving me a feel of confidence that it has separated out the different cases, and chosen a protocol suitable for each. I tried to go back to the early draft on flow control requirements < http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01 >, and I was not impressed.

I have quoted below the various sentences in the draft that state what flow control is believed to be for. Below that, I have attempted to crystalize out the different concepts, each of which I have tagged within the quotes.

* 2. HTTP/2 Protocol Overview says
  "Flow control and prioritization ensure that it is possible to efficiently use multiplexed streams. [Y]
   Flow control (Section 5.2) helps to ensure that only data that can be used by a receiver is transmitted. [X]"
* 5.2. Flow Control says:
  "Using streams for multiplexing introduces contention over use of the TCP connection [X], resulting in blocked streams [Z]. A flow control scheme ensures that streams on the same connection do not destructively interfere with each other [Z]."
* 5.2.2. Appropriate Use of Flow Control
"  Flow control is defined to protect endpoints that are operating under
   resource constraints.  For example, a proxy needs to share memory
   between many connections, and also might have a slow upstream
   connection and a fast downstream one [Y].  Flow control addresses cases
   where the receiver is unable to process data on one stream, yet wants
   to continue to process other streams in the same connection [X]."
"  Deployments with constrained resources (for example, memory) can
   employ flow control to limit the amount of memory a peer can consume. [Y]

Each requirement has been tagged as follows:
[X] Notification of the receiver's changing utility for each stream
[Y] Prioritisation of streams due to contention over the streaming capacity available to the whole connection.
[Z] Ensuring one stream is not blocked by another.

[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y] sounds more like optimisation across a continuous spectrum.


Regards



Bob

________________________________________________________________
Bob Briscoe,                                                  BT

________________________________________________________________
Bob Briscoe,                                                  BT


Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Roberto Peon-2
In reply to this post by Bob Briscoe
If you have a multiplexed protocol, and you want proxy or loadbalance (basically true for most larger sites), then you'll be met with a rate mismatch where a sender on either side is sending faster than the receiver is reading. You are then presented with a few options:
  1) drop the connection and lose all the streams in progress
  2) drop the stream/data and lose any progress on that stream
  3) pause (i.e. head of line block) for an indeterminate amount of time until that stream drains
  4) exert flow control.
  5) infinitely buffer (i.e. OOM and lose all connections for all clients)

Of these, #4 seemed the best choice to keep the flows flowing.


I'd have loved to see the "blocked because of flow control" thing become a reality and thus allow autotuning, but I haven't seen any moaning about the lack of it. That implies either that stuff works fine, or that we're not paying close enough attention.

-=R

On Tue, Mar 10, 2015 at 1:31 PM, Bob Briscoe <[hidden email]> wrote:
HTTP/2 folks,

I know extensibility had already been discussed and put to bed, so the WG is entitled to rule out opening healed wounds.

But have points like those I've made about flow control been raised before? Please argue. I may be wrong. Discussion can go on in parallel to the RFC publication process, even tho the process doesn't /require/ you to talk to me.

If I'm right, then implementers are being mandated to write complex flow control code, when it might have little bearing on the performance benefits measured for http/2.

Even if I'm right, and the WG goes ahead anyway, /I/ will understand. My review came in after your deadline.

However, bear in mind that the Webosphere might not be so forgiving. If h2 goes ahead when potential problems have been identified, it could get a bad reputation simply due to the uncertainty, just when you want more people to take it up and try it out. Given you've put in a few person-years of effort, I would have thought you would not want to risk a reputation flop.

I'm trying to help - I just can't go any faster.


Bob


At 14:43 06/03/2015, Bob Briscoe wrote:
HTTP/2 folks,

As I said, consider this as a late review from a clueful but fresh pair of eyes.
My main concerns with the draft are:
* extensibility (previous posting)
* flow control (this posting - apologies for the length - I've tried to explain properly)
* numerous open issues left dangling (see subsequent postings)

===HTTP/2 FLOW CONTROL===

The term 'window' as used throughout is incorrect and highly confusing, in:
* 'flow control window' (44 occurrences),
* 'initial window size' (5),
* or just 'window size' (8)
* and the terms WINDOW_UPDATE and SETTINGS_INITIAL_WINDOW.

The HTTP/2 WINDOW_UPDATE mechanism constrains HTTP/2 to use only credit-based flow control, not window-based. At one point, it actually says it is credit-based (in flow control principle #2 < https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-5.2.1 > ), but otherwise it incorrectly uses the term window.

This is not just an issue of terminology. The more I re-read the flow control sections the more I became convinced that this terminology is not just /confusing/, rather it's evidence of /confusion/. It raises the questions
* "Is HTTP/2 capable of the flow control it says it's capable of?"
* "What type of flow-control protocol ought HTTP/2 to be capable of?"
* "Can the WINDOW_UPDATE frame support the flow-control that HTTP/2 needs?"

To address these questions, it may help if I separate the two different cases HTTP/2 flow control attempts to cover (my own separation, not from the draft):

a) Intermediate buffer control
Here, a stream's flow enters /and/ leaves a buffer (e.g. at the app-layer of an intermediate node).

b) Flow control by the ultimate client app.
Here flow never releases memory (at least not during the life of the connection). The flow is solely consuming more and more memory (e.g. data being rendered into a client app's memory).

==a) Intermediate buffer control==
For this, sliding window-based flow control would be appropriate, because the goal is to keep the e2e pipeline full without wasting buffer.

Let me prove HTTP/2 cannot do window flow control. For window flow control, the sender needs to be able to advance both the leading and trailing edges of the window. In the draft:
* WINDOW_UPDATE frames can only advance the leading edge of a 'window' (and they are constrained to positive values).
* To advance the trailing edge, window flow control would need a continuous stream of acknowledgements back to the sender (like TCP). The draft does not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at the transport layer, so the sending app-layer cannot advance the trailing edge of a 'window'.

So the protocol can only support credit-based flow control. It is incapable of supporting window flow control.

Next, I don't understand how a receiver can set the credit in 'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to answer the question "How much more can I send than I have seen ACK'd?" that would be easy. But because the protocol is restricted to credit, the sender needs the receiver to answer the much harder open-ended question, "How much more can I send?" So the sender needs the receiver to know how many ACKs the sender has seen, but neither of them know that.

The receiver can try, by taking a guess at the bandwidth-delay product, and adjusting the guess up or down, depending on whether its buffer is growing or shrinking. But this only works if the unknown bandwidth-delay product stays constant.

However, BDP will usually be highly variable, as other streams come and go. So, in the time it takes to get a good estimate of the per-stream BDP, it will probably have changed radically, or the stream will most likely have finished anyway. This is why TCP bases flow control on a window, not credit. By complementing window updates with ACK stream info, a TCP sender has sufficient info to control the flow.

The draft is indeed correct when it says:
"   this can lead to suboptimal use of available
   network resources if flow control is enabled without knowledge of the
   bandwidth-delay product (see [RFC7323]).
"
Was this meant to be a veiled criticism of the protocol's own design? A credit-based flow control protocol like that in the draft does not provide sufficient information for either end to estimate the bandwidth-delay product, given it will be varying rapidly.

==b) Control by the ultimate client app==
For this case, I believe neither window nor credit-based flow control is appropriate:
* There is no memory management issue at the client end - even if there's a separate HTTP/2 layer of memory between TCP and the app, it would be pointless to limit the memory used by HTTP/2, because the data is still going to sit in the same user-space memory (or at least about the same amount of memory) when HTTP/2 passes it over for rendering.
* Nonetheless, the receiving client does need to send messages to the sender to supplement stream priorities, by notifying when the state of the receiving application has changed (e.g. if the user's focus switches from one browser tab to another).
* However, credit-based flow control would be very sluggish for such control, because credit cannot be taken back once it has been given (except HTTP/2 allows SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure that hits all streams together).

==Flow control problem summary==

With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.

==Towards a solution?==

I think 'type-a' flow control (for intermediate buffer control) does not need to be at stream-granularity. Indeed, I suspect a proxy could control its app-layer buffering by controlling the receive window of the incoming TCP connection. Has anyone assessed whether this would be sufficient?

I can understand the need for 'type-b' per-stream flow control (by the ultimate client endpoint). Perhaps it would be useful for the receiver to emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream PRIORITY would be sufficient? Either would minimise the response time to a half round trip. Whereas credit flow-control will be much more sluggish (see 'Flow control problem summary').

Either approach would correctly propagate e2e. An intermediate node would naturally tend to prioritise incoming streams that fed into prioritised outgoing streams, so priority updates would tend to propagate from the ultimate receiver, through intermediate nodes, up to the ultimate sender.

==Flow control coverage==

The draft exempts all TCP payload bytes from flow control except HTTP/2 data frames. No rationale is given for this decision. The draft says it's important to manage per-stream memory, then it exempts all the frame types except data, even tho each byte of a non-data frame consumes no less memory than a byte of a data frame.

What message does this put out? "Flow control is not important for one type of bytes with unlimited total size, but flow control is so important that it has to be mandatory for the other type of bytes."

It is certainly critical that WINDOW_UPDATE messages are not covered by flow control, otherwise there would be a real risk of deadlock. It might be that there are dependencies on other frame types that would lead to a dependency loop and deadlock. It would be good to know what the rationale behind these rules was.

==Theory?==

I am concerned that HTTP/2 flow control may have entered new theoretical territory, without suitable proof of safety. The only reassurance we have is one implementation of a flow control algorithm (SPDY), and the anecdotal non-evidence that no-one using SPDY has noticed a deadlock yet (however, is anyone monitoring for deadlocks?).

Whereas SPDY has been an existence proof that an approach like http/2 'works', so far all the flow control algos have been pretty much identical (I think that's true?). I am concerned that the draft takes the InterWeb into uncharted waters, because it allows unconstrained diversity in flow control algos, which is an untested degree of freedom.

The only constraints the draft sets are:
* per-stream flow control is mandatory
* the only protocol message for flow control algos to use is the WINDOW_UPDATE credit message, which cannot be negative
* no constraints on flow control algorithms.
* and all this must work within the outer flow control constraints of TCP.

Some algos might use priority messages to make flow control assumptions. While other algos might associate PRI and WINDOW_UPDATE with different meanings. What confidence do we have that everyone's optimisation algorithms will interoperate? Do we know there will not be certain types of application where deadlock is likely?

"   When using flow
   control, the receiver MUST read from the TCP receive buffer in a
   timely fashion.  Failure to do so could lead to a deadlock when
   critical frames, such as WINDOW_UPDATE, are not read and acted upon.
"
I've been convinced (offlist) that deadlock will not occur as long as the app consumes data 'greedily' from TCP. That has since been articulated in the above normative text. But how sure can we be that every implementer's different interpretations of 'timely' will still prevent deadlock?

Until a good autotuning algorithm for TCP receive window management was developed, good window management code was nearly non-existent. Managing hundreds of interdependent stream buffers is a much harder problem. But implementers are being allowed to just 'Go forth and innovate'. This might work if everyone copies available open source algo(s). But they might not, and they don't have to.

This all seems like 'flying by the seat of the pants'.

==Mandatory Flow Control? ==

"      3. [...] A sender
       MUST respect flow control limits imposed by a receiver."
This ought to be a 'SHOULD' because it is contradicted later - if settings change.

"   6.  Flow control cannot be disabled."
Also effectively contradicted half a page later:
"   Deployments that do not require this capability can advertise a flow
   control window of the maximum size (2^31-1), and by maintaining this
   window by sending a WINDOW_UPDATE frame when any data is received.
   This effectively disables flow control for that receiver."

And contradicted in the definition of half closed (remote):
"  half closed (remote):
      [...] an endpoint is no longer
      obligated to maintain a receiver flow control window.
"

And contradicted in 8.3. The CONNECT Method, which says:
"  Frame types other than DATA
   or stream management frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY)
   MUST NOT be sent on a connected stream, and MUST be treated as a
   stream error (Section 5.4.2) if received.
"

Why is flow control so important that it's mandatory, but so unimportant that you MUST NOT do it when using TLS e2e?

Going back to the earlier quote about using the max window size, it seems perverse for the spec to require endpoints to go through the motions of flow control, even if they arrange for it to affect nothing, but to still require implementation complexity and bandwidth waste with a load of redundant WINDOW_UPDATE frames.

HTTP is used on a wide range of devices, down to the very small and challenged. HTTP/2 might be desirable in such cases, because of the improved efficiency (e.g. header compression), but in many cases the stream model may not be complex enough to need stream flow control.

So why not make flow control optional on the receiving side, but mandatory to implement on the sending side? Then an implementation could have no machinery for tuning window sizes, but it would respond correctly to those set by the other end, which requires much simpler code.

If a receiving implemention chose not to do stream flow control, it could still control flow at the connection (stream 0) level, or at least at the TCP level.


==Inefficiency?==

5.2. Flow Control
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.

==Flow Control Requirements===

I'm not convinced that clear understanding of flow control requirements has driven flow control design decisions.

The draft states various needs for flow-control without giving me a feel of confidence that it has separated out the different cases, and chosen a protocol suitable for each. I tried to go back to the early draft on flow control requirements < http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01 >, and I was not impressed.

I have quoted below the various sentences in the draft that state what flow control is believed to be for. Below that, I have attempted to crystalize out the different concepts, each of which I have tagged within the quotes.

* 2. HTTP/2 Protocol Overview says
  "Flow control and prioritization ensure that it is possible to efficiently use multiplexed streams. [Y]
   Flow control (Section 5.2) helps to ensure that only data that can be used by a receiver is transmitted. [X]"
* 5.2. Flow Control says:
  "Using streams for multiplexing introduces contention over use of the TCP connection [X], resulting in blocked streams [Z]. A flow control scheme ensures that streams on the same connection do not destructively interfere with each other [Z]."
* 5.2.2. Appropriate Use of Flow Control
"  Flow control is defined to protect endpoints that are operating under
   resource constraints.  For example, a proxy needs to share memory
   between many connections, and also might have a slow upstream
   connection and a fast downstream one [Y].  Flow control addresses cases
   where the receiver is unable to process data on one stream, yet wants
   to continue to process other streams in the same connection [X]."
"  Deployments with constrained resources (for example, memory) can
   employ flow control to limit the amount of memory a peer can consume. [Y]

Each requirement has been tagged as follows:
[X] Notification of the receiver's changing utility for each stream
[Y] Prioritisation of streams due to contention over the streaming capacity available to the whole connection.
[Z] Ensuring one stream is not blocked by another.

[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y] sounds more like optimisation across a continuous spectrum.


Regards



Bob

________________________________________________________________
Bob Briscoe,                                                  BT

________________________________________________________________
Bob Briscoe,                                                  BT


Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Greg Wilkins-3

Roberto,

nice summation.

Note that #3 is almost an OK option in many situations.  When sending multiple streams S->C, each stream consumer is likely to progress at the same rate (just moving stream into memory) or at least faster than the network (when moving stream to disk).   In such situations allowing the multiplex streams to flow at maximum allowed by TCP flow control would not directly introduce any HOL blocking issues.  So long as large streams do not prevent small streams progressing, the server can send interleaved frames at a rate that will saturate the available bandwidth.

The situation is not the same C->S where large uploads and other request content can be consumed at widely diverse rates and with just multiplexing if a large upload was allowed to hit the TCP flow control, then it could HOL block other small requests that should have been allowed to proceed.   So #4 is required for C->S.   But for that to work, we need the control frames sent in the other direction not to be HOL-blocked, which then rules out #3 for S->C!

It all seams reasonable and I can't think of what could be done better given the basic context of the problem (other than not incorrectly call it a "window"),  but it does leave us with a solution that only works as designed if it never runs at the maximum throughput available of the available TCP channel.  This might not be a problem if we routinely get close to the maximum available throughput, but experience may show that there are scenarios where we fall short of near optimal bandwidth utilisation.     It's something that we have to watch for carefully and luckily HTTP/1.1 is not going away any time soon, so we have an alternative for any such scenarios discovered.

regards








On 11 March 2015 at 12:38, Roberto Peon <[hidden email]> wrote:
If you have a multiplexed protocol, and you want proxy or loadbalance (basically true for most larger sites), then you'll be met with a rate mismatch where a sender on either side is sending faster than the receiver is reading. You are then presented with a few options:
  1) drop the connection and lose all the streams in progress
  2) drop the stream/data and lose any progress on that stream
  3) pause (i.e. head of line block) for an indeterminate amount of time until that stream drains
  4) exert flow control.
  5) infinitely buffer (i.e. OOM and lose all connections for all clients)

Of these, #4 seemed the best choice to keep the flows flowing.


I'd have loved to see the "blocked because of flow control" thing become a reality and thus allow autotuning, but I haven't seen any moaning about the lack of it. That implies either that stuff works fine, or that we're not paying close enough attention.

-=R

On Tue, Mar 10, 2015 at 1:31 PM, Bob Briscoe <[hidden email]> wrote:
HTTP/2 folks,

I know extensibility had already been discussed and put to bed, so the WG is entitled to rule out opening healed wounds.

But have points like those I've made about flow control been raised before? Please argue. I may be wrong. Discussion can go on in parallel to the RFC publication process, even tho the process doesn't /require/ you to talk to me.

If I'm right, then implementers are being mandated to write complex flow control code, when it might have little bearing on the performance benefits measured for http/2.

Even if I'm right, and the WG goes ahead anyway, /I/ will understand. My review came in after your deadline.

However, bear in mind that the Webosphere might not be so forgiving. If h2 goes ahead when potential problems have been identified, it could get a bad reputation simply due to the uncertainty, just when you want more people to take it up and try it out. Given you've put in a few person-years of effort, I would have thought you would not want to risk a reputation flop.

I'm trying to help - I just can't go any faster.


Bob


At 14:43 06/03/2015, Bob Briscoe wrote:
HTTP/2 folks,

As I said, consider this as a late review from a clueful but fresh pair of eyes.
My main concerns with the draft are:
* extensibility (previous posting)
* flow control (this posting - apologies for the length - I've tried to explain properly)
* numerous open issues left dangling (see subsequent postings)

===HTTP/2 FLOW CONTROL===

The term 'window' as used throughout is incorrect and highly confusing, in:
* 'flow control window' (44 occurrences),
* 'initial window size' (5),
* or just 'window size' (8)
* and the terms WINDOW_UPDATE and SETTINGS_INITIAL_WINDOW.

The HTTP/2 WINDOW_UPDATE mechanism constrains HTTP/2 to use only credit-based flow control, not window-based. At one point, it actually says it is credit-based (in flow control principle #2 < https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-5.2.1 > ), but otherwise it incorrectly uses the term window.

This is not just an issue of terminology. The more I re-read the flow control sections the more I became convinced that this terminology is not just /confusing/, rather it's evidence of /confusion/. It raises the questions
* "Is HTTP/2 capable of the flow control it says it's capable of?"
* "What type of flow-control protocol ought HTTP/2 to be capable of?"
* "Can the WINDOW_UPDATE frame support the flow-control that HTTP/2 needs?"

To address these questions, it may help if I separate the two different cases HTTP/2 flow control attempts to cover (my own separation, not from the draft):

a) Intermediate buffer control
Here, a stream's flow enters /and/ leaves a buffer (e.g. at the app-layer of an intermediate node).

b) Flow control by the ultimate client app.
Here flow never releases memory (at least not during the life of the connection). The flow is solely consuming more and more memory (e.g. data being rendered into a client app's memory).

==a) Intermediate buffer control==
For this, sliding window-based flow control would be appropriate, because the goal is to keep the e2e pipeline full without wasting buffer.

Let me prove HTTP/2 cannot do window flow control. For window flow control, the sender needs to be able to advance both the leading and trailing edges of the window. In the draft:
* WINDOW_UPDATE frames can only advance the leading edge of a 'window' (and they are constrained to positive values).
* To advance the trailing edge, window flow control would need a continuous stream of acknowledgements back to the sender (like TCP). The draft does not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at the transport layer, so the sending app-layer cannot advance the trailing edge of a 'window'.

So the protocol can only support credit-based flow control. It is incapable of supporting window flow control.

Next, I don't understand how a receiver can set the credit in 'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to answer the question "How much more can I send than I have seen ACK'd?" that would be easy. But because the protocol is restricted to credit, the sender needs the receiver to answer the much harder open-ended question, "How much more can I send?" So the sender needs the receiver to know how many ACKs the sender has seen, but neither of them know that.

The receiver can try, by taking a guess at the bandwidth-delay product, and adjusting the guess up or down, depending on whether its buffer is growing or shrinking. But this only works if the unknown bandwidth-delay product stays constant.

However, BDP will usually be highly variable, as other streams come and go. So, in the time it takes to get a good estimate of the per-stream BDP, it will probably have changed radically, or the stream will most likely have finished anyway. This is why TCP bases flow control on a window, not credit. By complementing window updates with ACK stream info, a TCP sender has sufficient info to control the flow.

The draft is indeed correct when it says:
"   this can lead to suboptimal use of available
   network resources if flow control is enabled without knowledge of the
   bandwidth-delay product (see [RFC7323]).
"
Was this meant to be a veiled criticism of the protocol's own design? A credit-based flow control protocol like that in the draft does not provide sufficient information for either end to estimate the bandwidth-delay product, given it will be varying rapidly.

==b) Control by the ultimate client app==
For this case, I believe neither window nor credit-based flow control is appropriate:
* There is no memory management issue at the client end - even if there's a separate HTTP/2 layer of memory between TCP and the app, it would be pointless to limit the memory used by HTTP/2, because the data is still going to sit in the same user-space memory (or at least about the same amount of memory) when HTTP/2 passes it over for rendering.
* Nonetheless, the receiving client does need to send messages to the sender to supplement stream priorities, by notifying when the state of the receiving application has changed (e.g. if the user's focus switches from one browser tab to another).
* However, credit-based flow control would be very sluggish for such control, because credit cannot be taken back once it has been given (except HTTP/2 allows SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure that hits all streams together).

==Flow control problem summary==

With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.

==Towards a solution?==

I think 'type-a' flow control (for intermediate buffer control) does not need to be at stream-granularity. Indeed, I suspect a proxy could control its app-layer buffering by controlling the receive window of the incoming TCP connection. Has anyone assessed whether this would be sufficient?

I can understand the need for 'type-b' per-stream flow control (by the ultimate client endpoint). Perhaps it would be useful for the receiver to emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream PRIORITY would be sufficient? Either would minimise the response time to a half round trip. Whereas credit flow-control will be much more sluggish (see 'Flow control problem summary').

Either approach would correctly propagate e2e. An intermediate node would naturally tend to prioritise incoming streams that fed into prioritised outgoing streams, so priority updates would tend to propagate from the ultimate receiver, through intermediate nodes, up to the ultimate sender.

==Flow control coverage==

The draft exempts all TCP payload bytes from flow control except HTTP/2 data frames. No rationale is given for this decision. The draft says it's important to manage per-stream memory, then it exempts all the frame types except data, even tho each byte of a non-data frame consumes no less memory than a byte of a data frame.

What message does this put out? "Flow control is not important for one type of bytes with unlimited total size, but flow control is so important that it has to be mandatory for the other type of bytes."

It is certainly critical that WINDOW_UPDATE messages are not covered by flow control, otherwise there would be a real risk of deadlock. It might be that there are dependencies on other frame types that would lead to a dependency loop and deadlock. It would be good to know what the rationale behind these rules was.

==Theory?==

I am concerned that HTTP/2 flow control may have entered new theoretical territory, without suitable proof of safety. The only reassurance we have is one implementation of a flow control algorithm (SPDY), and the anecdotal non-evidence that no-one using SPDY has noticed a deadlock yet (however, is anyone monitoring for deadlocks?).

Whereas SPDY has been an existence proof that an approach like http/2 'works', so far all the flow control algos have been pretty much identical (I think that's true?). I am concerned that the draft takes the InterWeb into uncharted waters, because it allows unconstrained diversity in flow control algos, which is an untested degree of freedom.

The only constraints the draft sets are:
* per-stream flow control is mandatory
* the only protocol message for flow control algos to use is the WINDOW_UPDATE credit message, which cannot be negative
* no constraints on flow control algorithms.
* and all this must work within the outer flow control constraints of TCP.

Some algos might use priority messages to make flow control assumptions. While other algos might associate PRI and WINDOW_UPDATE with different meanings. What confidence do we have that everyone's optimisation algorithms will interoperate? Do we know there will not be certain types of application where deadlock is likely?

"   When using flow
   control, the receiver MUST read from the TCP receive buffer in a
   timely fashion.  Failure to do so could lead to a deadlock when
   critical frames, such as WINDOW_UPDATE, are not read and acted upon.
"
I've been convinced (offlist) that deadlock will not occur as long as the app consumes data 'greedily' from TCP. That has since been articulated in the above normative text. But how sure can we be that every implementer's different interpretations of 'timely' will still prevent deadlock?

Until a good autotuning algorithm for TCP receive window management was developed, good window management code was nearly non-existent. Managing hundreds of interdependent stream buffers is a much harder problem. But implementers are being allowed to just 'Go forth and innovate'. This might work if everyone copies available open source algo(s). But they might not, and they don't have to.

This all seems like 'flying by the seat of the pants'.

==Mandatory Flow Control? ==

"      3. [...] A sender
       MUST respect flow control limits imposed by a receiver."
This ought to be a 'SHOULD' because it is contradicted later - if settings change.

"   6.  Flow control cannot be disabled."
Also effectively contradicted half a page later:
"   Deployments that do not require this capability can advertise a flow
   control window of the maximum size (2^31-1), and by maintaining this
   window by sending a WINDOW_UPDATE frame when any data is received.
   This effectively disables flow control for that receiver."

And contradicted in the definition of half closed (remote):
"  half closed (remote):
      [...] an endpoint is no longer
      obligated to maintain a receiver flow control window.
"

And contradicted in 8.3. The CONNECT Method, which says:
"  Frame types other than DATA
   or stream management frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY)
   MUST NOT be sent on a connected stream, and MUST be treated as a
   stream error (Section 5.4.2) if received.
"

Why is flow control so important that it's mandatory, but so unimportant that you MUST NOT do it when using TLS e2e?

Going back to the earlier quote about using the max window size, it seems perverse for the spec to require endpoints to go through the motions of flow control, even if they arrange for it to affect nothing, but to still require implementation complexity and bandwidth waste with a load of redundant WINDOW_UPDATE frames.

HTTP is used on a wide range of devices, down to the very small and challenged. HTTP/2 might be desirable in such cases, because of the improved efficiency (e.g. header compression), but in many cases the stream model may not be complex enough to need stream flow control.

So why not make flow control optional on the receiving side, but mandatory to implement on the sending side? Then an implementation could have no machinery for tuning window sizes, but it would respond correctly to those set by the other end, which requires much simpler code.

If a receiving implemention chose not to do stream flow control, it could still control flow at the connection (stream 0) level, or at least at the TCP level.


==Inefficiency?==

5.2. Flow Control
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.

==Flow Control Requirements===

I'm not convinced that clear understanding of flow control requirements has driven flow control design decisions.

The draft states various needs for flow-control without giving me a feel of confidence that it has separated out the different cases, and chosen a protocol suitable for each. I tried to go back to the early draft on flow control requirements < http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01 >, and I was not impressed.

I have quoted below the various sentences in the draft that state what flow control is believed to be for. Below that, I have attempted to crystalize out the different concepts, each of which I have tagged within the quotes.

* 2. HTTP/2 Protocol Overview says
  "Flow control and prioritization ensure that it is possible to efficiently use multiplexed streams. [Y]
   Flow control (Section 5.2) helps to ensure that only data that can be used by a receiver is transmitted. [X]"
* 5.2. Flow Control says:
  "Using streams for multiplexing introduces contention over use of the TCP connection [X], resulting in blocked streams [Z]. A flow control scheme ensures that streams on the same connection do not destructively interfere with each other [Z]."
* 5.2.2. Appropriate Use of Flow Control
"  Flow control is defined to protect endpoints that are operating under
   resource constraints.  For example, a proxy needs to share memory
   between many connections, and also might have a slow upstream
   connection and a fast downstream one [Y].  Flow control addresses cases
   where the receiver is unable to process data on one stream, yet wants
   to continue to process other streams in the same connection [X]."
"  Deployments with constrained resources (for example, memory) can
   employ flow control to limit the amount of memory a peer can consume. [Y]

Each requirement has been tagged as follows:
[X] Notification of the receiver's changing utility for each stream
[Y] Prioritisation of streams due to contention over the streaming capacity available to the whole connection.
[Z] Ensuring one stream is not blocked by another.

[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y] sounds more like optimisation across a continuous spectrum.


Regards



Bob

________________________________________________________________
Bob Briscoe,                                                  BT

________________________________________________________________
Bob Briscoe,                                                  BT





--
Greg Wilkins <[hidden email]>  @  Webtide - an Intalio subsidiary
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.
Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Roberto Peon-2
Agreed.
-=R

On Wed, Mar 11, 2015 at 6:42 PM, Greg Wilkins <[hidden email]> wrote:

Roberto,

nice summation.

Note that #3 is almost an OK option in many situations.  When sending multiple streams S->C, each stream consumer is likely to progress at the same rate (just moving stream into memory) or at least faster than the network (when moving stream to disk).   In such situations allowing the multiplex streams to flow at maximum allowed by TCP flow control would not directly introduce any HOL blocking issues.  So long as large streams do not prevent small streams progressing, the server can send interleaved frames at a rate that will saturate the available bandwidth.

The situation is not the same C->S where large uploads and other request content can be consumed at widely diverse rates and with just multiplexing if a large upload was allowed to hit the TCP flow control, then it could HOL block other small requests that should have been allowed to proceed.   So #4 is required for C->S.   But for that to work, we need the control frames sent in the other direction not to be HOL-blocked, which then rules out #3 for S->C!

It all seams reasonable and I can't think of what could be done better given the basic context of the problem (other than not incorrectly call it a "window"),  but it does leave us with a solution that only works as designed if it never runs at the maximum throughput available of the available TCP channel.  This might not be a problem if we routinely get close to the maximum available throughput, but experience may show that there are scenarios where we fall short of near optimal bandwidth utilisation.     It's something that we have to watch for carefully and luckily HTTP/1.1 is not going away any time soon, so we have an alternative for any such scenarios discovered.

regards








On 11 March 2015 at 12:38, Roberto Peon <[hidden email]> wrote:
If you have a multiplexed protocol, and you want proxy or loadbalance (basically true for most larger sites), then you'll be met with a rate mismatch where a sender on either side is sending faster than the receiver is reading. You are then presented with a few options:
  1) drop the connection and lose all the streams in progress
  2) drop the stream/data and lose any progress on that stream
  3) pause (i.e. head of line block) for an indeterminate amount of time until that stream drains
  4) exert flow control.
  5) infinitely buffer (i.e. OOM and lose all connections for all clients)

Of these, #4 seemed the best choice to keep the flows flowing.


I'd have loved to see the "blocked because of flow control" thing become a reality and thus allow autotuning, but I haven't seen any moaning about the lack of it. That implies either that stuff works fine, or that we're not paying close enough attention.

-=R

On Tue, Mar 10, 2015 at 1:31 PM, Bob Briscoe <[hidden email]> wrote:
HTTP/2 folks,

I know extensibility had already been discussed and put to bed, so the WG is entitled to rule out opening healed wounds.

But have points like those I've made about flow control been raised before? Please argue. I may be wrong. Discussion can go on in parallel to the RFC publication process, even tho the process doesn't /require/ you to talk to me.

If I'm right, then implementers are being mandated to write complex flow control code, when it might have little bearing on the performance benefits measured for http/2.

Even if I'm right, and the WG goes ahead anyway, /I/ will understand. My review came in after your deadline.

However, bear in mind that the Webosphere might not be so forgiving. If h2 goes ahead when potential problems have been identified, it could get a bad reputation simply due to the uncertainty, just when you want more people to take it up and try it out. Given you've put in a few person-years of effort, I would have thought you would not want to risk a reputation flop.

I'm trying to help - I just can't go any faster.


Bob


At 14:43 06/03/2015, Bob Briscoe wrote:
HTTP/2 folks,

As I said, consider this as a late review from a clueful but fresh pair of eyes.
My main concerns with the draft are:
* extensibility (previous posting)
* flow control (this posting - apologies for the length - I've tried to explain properly)
* numerous open issues left dangling (see subsequent postings)

===HTTP/2 FLOW CONTROL===

The term 'window' as used throughout is incorrect and highly confusing, in:
* 'flow control window' (44 occurrences),
* 'initial window size' (5),
* or just 'window size' (8)
* and the terms WINDOW_UPDATE and SETTINGS_INITIAL_WINDOW.

The HTTP/2 WINDOW_UPDATE mechanism constrains HTTP/2 to use only credit-based flow control, not window-based. At one point, it actually says it is credit-based (in flow control principle #2 < https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-5.2.1 > ), but otherwise it incorrectly uses the term window.

This is not just an issue of terminology. The more I re-read the flow control sections the more I became convinced that this terminology is not just /confusing/, rather it's evidence of /confusion/. It raises the questions
* "Is HTTP/2 capable of the flow control it says it's capable of?"
* "What type of flow-control protocol ought HTTP/2 to be capable of?"
* "Can the WINDOW_UPDATE frame support the flow-control that HTTP/2 needs?"

To address these questions, it may help if I separate the two different cases HTTP/2 flow control attempts to cover (my own separation, not from the draft):

a) Intermediate buffer control
Here, a stream's flow enters /and/ leaves a buffer (e.g. at the app-layer of an intermediate node).

b) Flow control by the ultimate client app.
Here flow never releases memory (at least not during the life of the connection). The flow is solely consuming more and more memory (e.g. data being rendered into a client app's memory).

==a) Intermediate buffer control==
For this, sliding window-based flow control would be appropriate, because the goal is to keep the e2e pipeline full without wasting buffer.

Let me prove HTTP/2 cannot do window flow control. For window flow control, the sender needs to be able to advance both the leading and trailing edges of the window. In the draft:
* WINDOW_UPDATE frames can only advance the leading edge of a 'window' (and they are constrained to positive values).
* To advance the trailing edge, window flow control would need a continuous stream of acknowledgements back to the sender (like TCP). The draft does not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at the transport layer, so the sending app-layer cannot advance the trailing edge of a 'window'.

So the protocol can only support credit-based flow control. It is incapable of supporting window flow control.

Next, I don't understand how a receiver can set the credit in 'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to answer the question "How much more can I send than I have seen ACK'd?" that would be easy. But because the protocol is restricted to credit, the sender needs the receiver to answer the much harder open-ended question, "How much more can I send?" So the sender needs the receiver to know how many ACKs the sender has seen, but neither of them know that.

The receiver can try, by taking a guess at the bandwidth-delay product, and adjusting the guess up or down, depending on whether its buffer is growing or shrinking. But this only works if the unknown bandwidth-delay product stays constant.

However, BDP will usually be highly variable, as other streams come and go. So, in the time it takes to get a good estimate of the per-stream BDP, it will probably have changed radically, or the stream will most likely have finished anyway. This is why TCP bases flow control on a window, not credit. By complementing window updates with ACK stream info, a TCP sender has sufficient info to control the flow.

The draft is indeed correct when it says:
"   this can lead to suboptimal use of available
   network resources if flow control is enabled without knowledge of the
   bandwidth-delay product (see [RFC7323]).
"
Was this meant to be a veiled criticism of the protocol's own design? A credit-based flow control protocol like that in the draft does not provide sufficient information for either end to estimate the bandwidth-delay product, given it will be varying rapidly.

==b) Control by the ultimate client app==
For this case, I believe neither window nor credit-based flow control is appropriate:
* There is no memory management issue at the client end - even if there's a separate HTTP/2 layer of memory between TCP and the app, it would be pointless to limit the memory used by HTTP/2, because the data is still going to sit in the same user-space memory (or at least about the same amount of memory) when HTTP/2 passes it over for rendering.
* Nonetheless, the receiving client does need to send messages to the sender to supplement stream priorities, by notifying when the state of the receiving application has changed (e.g. if the user's focus switches from one browser tab to another).
* However, credit-based flow control would be very sluggish for such control, because credit cannot be taken back once it has been given (except HTTP/2 allows SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure that hits all streams together).

==Flow control problem summary==

With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.

==Towards a solution?==

I think 'type-a' flow control (for intermediate buffer control) does not need to be at stream-granularity. Indeed, I suspect a proxy could control its app-layer buffering by controlling the receive window of the incoming TCP connection. Has anyone assessed whether this would be sufficient?

I can understand the need for 'type-b' per-stream flow control (by the ultimate client endpoint). Perhaps it would be useful for the receiver to emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream PRIORITY would be sufficient? Either would minimise the response time to a half round trip. Whereas credit flow-control will be much more sluggish (see 'Flow control problem summary').

Either approach would correctly propagate e2e. An intermediate node would naturally tend to prioritise incoming streams that fed into prioritised outgoing streams, so priority updates would tend to propagate from the ultimate receiver, through intermediate nodes, up to the ultimate sender.

==Flow control coverage==

The draft exempts all TCP payload bytes from flow control except HTTP/2 data frames. No rationale is given for this decision. The draft says it's important to manage per-stream memory, then it exempts all the frame types except data, even tho each byte of a non-data frame consumes no less memory than a byte of a data frame.

What message does this put out? "Flow control is not important for one type of bytes with unlimited total size, but flow control is so important that it has to be mandatory for the other type of bytes."

It is certainly critical that WINDOW_UPDATE messages are not covered by flow control, otherwise there would be a real risk of deadlock. It might be that there are dependencies on other frame types that would lead to a dependency loop and deadlock. It would be good to know what the rationale behind these rules was.

==Theory?==

I am concerned that HTTP/2 flow control may have entered new theoretical territory, without suitable proof of safety. The only reassurance we have is one implementation of a flow control algorithm (SPDY), and the anecdotal non-evidence that no-one using SPDY has noticed a deadlock yet (however, is anyone monitoring for deadlocks?).

Whereas SPDY has been an existence proof that an approach like http/2 'works', so far all the flow control algos have been pretty much identical (I think that's true?). I am concerned that the draft takes the InterWeb into uncharted waters, because it allows unconstrained diversity in flow control algos, which is an untested degree of freedom.

The only constraints the draft sets are:
* per-stream flow control is mandatory
* the only protocol message for flow control algos to use is the WINDOW_UPDATE credit message, which cannot be negative
* no constraints on flow control algorithms.
* and all this must work within the outer flow control constraints of TCP.

Some algos might use priority messages to make flow control assumptions. While other algos might associate PRI and WINDOW_UPDATE with different meanings. What confidence do we have that everyone's optimisation algorithms will interoperate? Do we know there will not be certain types of application where deadlock is likely?

"   When using flow
   control, the receiver MUST read from the TCP receive buffer in a
   timely fashion.  Failure to do so could lead to a deadlock when
   critical frames, such as WINDOW_UPDATE, are not read and acted upon.
"
I've been convinced (offlist) that deadlock will not occur as long as the app consumes data 'greedily' from TCP. That has since been articulated in the above normative text. But how sure can we be that every implementer's different interpretations of 'timely' will still prevent deadlock?

Until a good autotuning algorithm for TCP receive window management was developed, good window management code was nearly non-existent. Managing hundreds of interdependent stream buffers is a much harder problem. But implementers are being allowed to just 'Go forth and innovate'. This might work if everyone copies available open source algo(s). But they might not, and they don't have to.

This all seems like 'flying by the seat of the pants'.

==Mandatory Flow Control? ==

"      3. [...] A sender
       MUST respect flow control limits imposed by a receiver."
This ought to be a 'SHOULD' because it is contradicted later - if settings change.

"   6.  Flow control cannot be disabled."
Also effectively contradicted half a page later:
"   Deployments that do not require this capability can advertise a flow
   control window of the maximum size (2^31-1), and by maintaining this
   window by sending a WINDOW_UPDATE frame when any data is received.
   This effectively disables flow control for that receiver."

And contradicted in the definition of half closed (remote):
"  half closed (remote):
      [...] an endpoint is no longer
      obligated to maintain a receiver flow control window.
"

And contradicted in 8.3. The CONNECT Method, which says:
"  Frame types other than DATA
   or stream management frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY)
   MUST NOT be sent on a connected stream, and MUST be treated as a
   stream error (Section 5.4.2) if received.
"

Why is flow control so important that it's mandatory, but so unimportant that you MUST NOT do it when using TLS e2e?

Going back to the earlier quote about using the max window size, it seems perverse for the spec to require endpoints to go through the motions of flow control, even if they arrange for it to affect nothing, but to still require implementation complexity and bandwidth waste with a load of redundant WINDOW_UPDATE frames.

HTTP is used on a wide range of devices, down to the very small and challenged. HTTP/2 might be desirable in such cases, because of the improved efficiency (e.g. header compression), but in many cases the stream model may not be complex enough to need stream flow control.

So why not make flow control optional on the receiving side, but mandatory to implement on the sending side? Then an implementation could have no machinery for tuning window sizes, but it would respond correctly to those set by the other end, which requires much simpler code.

If a receiving implemention chose not to do stream flow control, it could still control flow at the connection (stream 0) level, or at least at the TCP level.


==Inefficiency?==

5.2. Flow Control
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.

==Flow Control Requirements===

I'm not convinced that clear understanding of flow control requirements has driven flow control design decisions.

The draft states various needs for flow-control without giving me a feel of confidence that it has separated out the different cases, and chosen a protocol suitable for each. I tried to go back to the early draft on flow control requirements < http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01 >, and I was not impressed.

I have quoted below the various sentences in the draft that state what flow control is believed to be for. Below that, I have attempted to crystalize out the different concepts, each of which I have tagged within the quotes.

* 2. HTTP/2 Protocol Overview says
  "Flow control and prioritization ensure that it is possible to efficiently use multiplexed streams. [Y]
   Flow control (Section 5.2) helps to ensure that only data that can be used by a receiver is transmitted. [X]"
* 5.2. Flow Control says:
  "Using streams for multiplexing introduces contention over use of the TCP connection [X], resulting in blocked streams [Z]. A flow control scheme ensures that streams on the same connection do not destructively interfere with each other [Z]."
* 5.2.2. Appropriate Use of Flow Control
"  Flow control is defined to protect endpoints that are operating under
   resource constraints.  For example, a proxy needs to share memory
   between many connections, and also might have a slow upstream
   connection and a fast downstream one [Y].  Flow control addresses cases
   where the receiver is unable to process data on one stream, yet wants
   to continue to process other streams in the same connection [X]."
"  Deployments with constrained resources (for example, memory) can
   employ flow control to limit the amount of memory a peer can consume. [Y]

Each requirement has been tagged as follows:
[X] Notification of the receiver's changing utility for each stream
[Y] Prioritisation of streams due to contention over the streaming capacity available to the whole connection.
[Z] Ensuring one stream is not blocked by another.

[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y] sounds more like optimisation across a continuous spectrum.


Regards



Bob

________________________________________________________________
Bob Briscoe,                                                  BT

________________________________________________________________
Bob Briscoe,                                                  BT





--
Greg Wilkins <[hidden email]>  @  Webtide - an Intalio subsidiary
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Bob Briscoe
In reply to this post by Patrick McManus-3
Patrick,

Thank you for taking the time to read my review carefully. I've been away from mail a few days, which should have allowed time for anything substantive to come from the people with SPDY experience.

We only had the Rob/Greg exchange which merely talked about what some theoretical thing called flow control might be useful for, rather than really addressing my concern that the credit-based protocol provided in h2 is unlikely to be able to provide control if it is needed. Nonetheless, Greg did summarise well the limited usefulness of h2's mechanism.

I should probably have provided a summary of my (long) review, which I try to do below.


I agree that there could be some use for the h2 credit-based protocol at the very start of a stream (in both the C->S and S->C PUSH cases you mentioned). And it might possibly be useful for the cases you mention about unsolicited data, which sound like they occur at the start too. Altho without knowing more about these DoS cases I'm not sure; RST_STREAM might have been sufficient.

However, once a stream has been allowed to open up it's rate, your observation that we mostly see very large windows is what I predicted. It demonstrates that the credit-based protocol does not have sufficient information to be useful to regulate flow. It is effectively being treated like the human appendix - something that no longer serves any purpose but you have to continually put in effort to keep it healthy otherwise it could stop the rest of the system from working.

For this reason, I questioned why flow control has been made mandatory. And I suggested instead that the credit-based flow control in h2 could be
i) mandatory for a data sender to respond to incoming WINDOW_UPDATEs (and therefore a data receiver can gracefully protect itself from DoS by discarding data that exceeds the credit it has previously made available)
ii) optional for a data receiver to emit WINDOW_UPDATEs (i.e. does not even have to implement this part of the code).


Bob

At 21:12 10/03/2015, Patrick McManus wrote:
Hi Bob - I think your comments are appreciated. Its just one of those things where people have dispersed to other things and aren't necessarily in a place to revisit all the ground work again at this stage for a new go round. It was in large part the operational feedback and needs of the google.com team, who has a lot of experience operating spdy at scale, that created the flow control provisions. hopefully those folks will chime in more authoritatively than my musings below:

I'm sure there is quite a bit to learn here - indeed poorly configured use of the window_update mechanism has been (predictably) a source of unintended bottlenecks during both spdy and h2 trials. The spec does try and highlight that there can be dragons here and implementations that don't need the features it can bring should provide essentially infinite credits to steer clear of them.

During the various trials I've seen h2 per stream flow control deployed successfully for a couple of use cases - both of them essentially deal with unsolicited data.

The primary one is essentially a more flexible version of h1's 100-continue. When a client presents a large message body (e.g. a file upload) a multiplexing server needs a way of saying "these are how many buffers I've got available while I figure out where I'm going to sink this incoming data (perhaps to another server I need to connect to)". Presenting this on a per-stream basis allows the server to limit one stream while another (with a distinct sink) can proceed independently.  IMO this value should represent resources available and should be independent of BDP. This is why in practice you see clients with extremely large stream windows - most circumstances just want the data to flow at line rate (as you describe) and aren't trying to invoke flow control. The firefox default window is 256MB per stream - that's not going to slow down the sender nor require frequent window_update generation.

The other use case is when the server pushes resources at a client without them being requested, which is a new feature of h2. This is conceptually similar to the server receiving a POST - suddenly there is a large amount of inbound data that the implementation might not have the resources to store completely. We can't just let TCP flow control take care of the situation because the TCP session is being multiplexed between multiple streams that need to be serviced. In this case the client accepts "some" of the stream based on policy and resource availability and can leave the stream in limbo until an event comes along that tells it to resume the transfer by issuing credits or reject it via cancel.

hth a least a bit.


On Tue, Mar 10, 2015 at 4:31 PM, Bob Briscoe <[hidden email]> wrote:
HTTP/2 folks,

I know extensibility had already been discussed and put to bed, so the WG is entitled to rule out opening healed wounds.

But have points like those I've made about flow control been raised before? Please argue. I may be wrong. Discussion can go on in parallel to the RFC publication process, even tho the process doesn't /require/ you to talk to me.

If I'm right, then implementers are being mandated to write complex flow control code, when it might have little bearing on the performance benefits measured for http/2.

Even if I'm right, and the WG goes ahead anyway, /I/ will understand. My review came in after your deadline.

However, bear in mind that the Webosphere might not be so forgiving. If h2 goes ahead when potential problems have been identified, it could get a bad reputation simply due to the uncertainty, just when you want more people to take it up and try it out. Given you've put in a few person-years of effort, I would have thought you would not want to risk a reputation flop.

I'm trying to help - I just can't go any faster.


Bob


At 14:43 06/03/2015, Bob Briscoe wrote:
HTTP/2 folks,

As I said, consider this as a late review from a clueful but fresh pair of eyes.
My main concerns with the draft are:
* extensibility (previous posting)
* flow control (this posting - apologies for the length - I've tried to explain properly)
* numerous open issues left dangling (see subsequent postings)

===HTTP/2 FLOW CONTROL===

The term 'window' as used throughout is incorrect and highly confusing, in:
* 'flow control window' (44 occurrences),
* 'initial window size' (5),
* or just 'window size' (8)
* and the terms WINDOW_UPDATE and SETTINGS_INITIAL_WINDOW.

The HTTP/2 WINDOW_UPDATE mechanism constrains HTTP/2 to use only credit-based flow control, not window-based. At one point, it actually says it is credit-based (in flow control principle #2 < https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-5.2.1 > ), but otherwise it incorrectly uses the term window.

This is not just an issue of terminology. The more I re-read the flow control sections the more I became convinced that this terminology is not just /confusing/, rather it's evidence of /confusion/. It raises the questions
* "Is HTTP/2 capable of the flow control it says it's capable of?"
* "What type of flow-control protocol ought HTTP/2 to be capable of?"
* "Can the WINDOW_UPDATE frame support the flow-control that HTTP/2 needs?"

To address these questions, it may help if I separate the two different cases HTTP/2 flow control attempts to cover (my own separation, not from the draft):

a) Intermediate buffer control
Here, a stream's flow enters /and/ leaves a buffer (e.g. at the app-layer of an intermediate node).

b) Flow control by the ultimate client app.
Here flow never releases memory (at least not during the life of the connection). The flow is solely consuming more and more memory (e.g. data being rendered into a client app's memory).

==a) Intermediate buffer control==
For this, sliding window-based flow control would be appropriate, because the goal is to keep the e2e pipeline full without wasting buffer.

Let me prove HTTP/2 cannot do window flow control. For window flow control, the sender needs to be able to advance both the leading and trailing edges of the window. In the draft:
* WINDOW_UPDATE frames can only advance the leading edge of a 'window' (and they are constrained to positive values).
* To advance the trailing edge, window flow control would need a continuous stream of acknowledgements back to the sender (like TCP). The draft does not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at the transport layer, so the sending app-layer cannot advance the trailing edge of a 'window'.

So the protocol can only support credit-based flow control. It is incapable of supporting window flow control.

Next, I don't understand how a receiver can set the credit in 'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to answer the question "How much more can I send than I have seen ACK'd?" that would be easy. But because the protocol is restricted to credit, the sender needs the receiver to answer the much harder open-ended question, "How much more can I send?" So the sender needs the receiver to know how many ACKs the sender has seen, but neither of them know that.

The receiver can try, by taking a guess at the bandwidth-delay product, and adjusting the guess up or down, depending on whether its buffer is growing or shrinking. But this only works if the unknown bandwidth-delay product stays constant.

However, BDP will usually be highly variable, as other streams come and go. So, in the time it takes to get a good estimate of the per-stream BDP, it will probably have changed radically, or the stream will most likely have finished anyway. This is why TCP bases flow control on a window, not credit. By complementing window updates with ACK stream info, a TCP sender has sufficient info to control the flow.

The draft is indeed correct when it says:
"   this can lead to suboptimal use of available
   network resources if flow control is enabled without knowledge of the
   bandwidth-delay product (see [RFC7323]).
"
Was this meant to be a veiled criticism of the protocol's own design? A credit-based flow control protocol like that in the draft does not provide sufficient information for either end to estimate the bandwidth-delay product, given it will be varying rapidly.

==b) Control by the ultimate client app==
For this case, I believe neither window nor credit-based flow control is appropriate:
* There is no memory management issue at the client end - even if there's a separate HTTP/2 layer of memory between TCP and the app, it would be pointless to limit the memory used by HTTP/2, because the data is still going to sit in the same user-space memory (or at least about the same amount of memory) when HTTP/2 passes it over for rendering.
* Nonetheless, the receiving client does need to send messages to the sender to supplement stream priorities, by notifying when the state of the receiving application has changed (e.g. if the user's focus switches from one browser tab to another).
* However, credit-based flow control would be very sluggish for such control, because credit cannot be taken back once it has been given (except HTTP/2 allows SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure that hits all streams together).

==Flow control problem summary==

With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.

==Towards a solution?==

I think 'type-a' flow control (for intermediate buffer control) does not need to be at stream-granularity. Indeed, I suspect a proxy could control its app-layer buffering by controlling the receive window of the incoming TCP connection. Has anyone assessed whether this would be sufficient?

I can understand the need for 'type-b' per-stream flow control (by the ultimate client endpoint). Perhaps it would be useful for the receiver to emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream PRIORITY would be sufficient? Either would minimise the response time to a half round trip. Whereas credit flow-control will be much more sluggish (see 'Flow control problem summary').

Either approach would correctly propagate e2e. An intermediate node would naturally tend to prioritise incoming streams that fed into prioritised outgoing streams, so priority updates would tend to propagate from the ultimate receiver, through intermediate nodes, up to the ultimate sender.

==Flow control coverage==

The draft exempts all TCP payload bytes from flow control except HTTP/2 data frames. No rationale is given for this decision. The draft says it's important to manage per-stream memory, then it exempts all the frame types except data, even tho each byte of a non-data frame consumes no less memory than a byte of a data frame.

What message does this put out? "Flow control is not important for one type of bytes with unlimited total size, but flow control is so important that it has to be mandatory for the other type of bytes."

It is certainly critical that WINDOW_UPDATE messages are not covered by flow control, otherwise there would be a real risk of deadlock. It might be that there are dependencies on other frame types that would lead to a dependency loop and deadlock. It would be good to know what the rationale behind these rules was.

==Theory?==

I am concerned that HTTP/2 flow control may have entered new theoretical territory, without suitable proof of safety. The only reassurance we have is one implementation of a flow control algorithm (SPDY), and the anecdotal non-evidence that no-one using SPDY has noticed a deadlock yet (however, is anyone monitoring for deadlocks?).

Whereas SPDY has been an existence proof that an approach like http/2 'works', so far all the flow control algos have been pretty much identical (I think that's true?). I am concerned that the draft takes the InterWeb into uncharted waters, because it allows unconstrained diversity in flow control algos, which is an untested degree of freedom.

The only constraints the draft sets are:
* per-stream flow control is mandatory
* the only protocol message for flow control algos to use is the WINDOW_UPDATE credit message, which cannot be negative
* no constraints on flow control algorithms.
* and all this must work within the outer flow control constraints of TCP.

Some algos might use priority messages to make flow control assumptions. While other algos might associate PRI and WINDOW_UPDATE with different meanings. What confidence do we have that everyone's optimisation algorithms will interoperate? Do we know there will not be certain types of application where deadlock is likely?

"   When using flow
   control, the receiver MUST read from the TCP receive buffer in a
   timely fashion.  Failure to do so could lead to a deadlock when
   critical frames, such as WINDOW_UPDATE, are not read and acted upon.
"
I've been convinced (offlist) that deadlock will not occur as long as the app consumes data 'greedily' from TCP. That has since been articulated in the above normative text. But how sure can we be that every implementer's different interpretations of 'timely' will still prevent deadlock?

Until a good autotuning algorithm for TCP receive window management was developed, good window management code was nearly non-existent. Managing hundreds of interdependent stream buffers is a much harder problem. But implementers are being allowed to just 'Go forth and innovate'. This might work if everyone copies available open source algo(s). But they might not, and they don't have to.

This all seems like 'flying by the seat of the pants'.

==Mandatory Flow Control? ==

"      3. [...] A sender
       MUST respect flow control limits imposed by a receiver."
This ought to be a 'SHOULD' because it is contradicted later - if settings change.

"   6.  Flow control cannot be disabled."
Also effectively contradicted half a page later:
"   Deployments that do not require this capability can advertise a flow
   control window of the maximum size (2^31-1), and by maintaining this
   window by sending a WINDOW_UPDATE frame when any data is received.
   This effectively disables flow control for that receiver."

And contradicted in the definition of half closed (remote):
"  half closed (remote):
      [...] an endpoint is no longer
      obligated to maintain a receiver flow control window.
"

And contradicted in 8.3. The CONNECT Method, which says:
"  Frame types other than DATA
   or stream management frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY)
   MUST NOT be sent on a connected stream, and MUST be treated as a
   stream error (Section 5.4.2) if received.
"

Why is flow control so important that it's mandatory, but so unimportant that you MUST NOT do it when using TLS e2e?

Going back to the earlier quote about using the max window size, it seems perverse for the spec to require endpoints to go through the motions of flow control, even if they arrange for it to affect nothing, but to still require implementation complexity and bandwidth waste with a load of redundant WINDOW_UPDATE frames.

HTTP is used on a wide range of devices, down to the very small and challenged. HTTP/2 might be desirable in such cases, because of the improved efficiency (e.g. header compression), but in many cases the stream model may not be complex enough to need stream flow control.

So why not make flow control optional on the receiving side, but mandatory to implement on the sending side? Then an implementation could have no machinery for tuning window sizes, but it would respond correctly to those set by the other end, which requires much simpler code.

If a receiving implemention chose not to do stream flow control, it could still control flow at the connection (stream 0) level, or at least at the TCP level.


==Inefficiency?==

5.2. Flow Control
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.

==Flow Control Requirements===

I'm not convinced that clear understanding of flow control requirements has driven flow control design decisions.

The draft states various needs for flow-control without giving me a feel of confidence that it has separated out the different cases, and chosen a protocol suitable for each. I tried to go back to the early draft on flow control requirements < http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01 >, and I was not impressed.

I have quoted below the various sentences in the draft that state what flow control is believed to be for. Below that, I have attempted to crystalize out the different concepts, each of which I have tagged within the quotes.

* 2. HTTP/2 Protocol Overview says
  "Flow control and prioritization ensure that it is possible to efficiently use multiplexed streams. [Y]
   Flow control (Section 5.2) helps to ensure that only data that can be used by a receiver is transmitted. [X]"
* 5.2. Flow Control says:
  "Using streams for multiplexing introduces contention over use of the TCP connection [X], resulting in blocked streams [Z]. A flow control scheme ensures that streams on the same connection do not destructively interfere with each other [Z]."
* 5.2.2. Appropriate Use of Flow Control
"  Flow control is defined to protect endpoints that are operating under
   resource constraints.  For example, a proxy needs to share memory
   between many connections, and also might have a slow upstream
   connection and a fast downstream one [Y].  Flow control addresses cases
   where the receiver is unable to process data on one stream, yet wants
   to continue to process other streams in the same connection [X]."
"  Deployments with constrained resources (for example, memory) can
   employ flow control to limit the amount of memory a peer can consume. [Y]

Each requirement has been tagged as follows:
[X] Notification of the receiver's changing utility for each stream
[Y] Prioritisation of streams due to contention over the streaming capacity available to the whole connection.
[Z] Ensuring one stream is not blocked by another.

[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y] sounds more like optimisation across a continuous spectrum.


Regards



Bob

________________________________________________________________
Bob Briscoe,                                                  BT

________________________________________________________________
Bob Briscoe,                                                  BT

________________________________________________________________
Bob Briscoe,                                                  BT

Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Jason Greene
Hi Bob,

I agree with you that HTTP/2 flow control is not useful for your a) use-case, intermediate buffer control. It is, however, necessary for the b) use case, memory management of an endpoint. 

An HTTP server is often divided into a core set of components which manage work scheduling and HTTP protocol aspects, as well as a set of independantly developed  “application” components, that  are often, but not necessarily,  provided by the end-user of the server. There is typically an abstraction between application code and server code, where the application is only ever aware of its stream. The stream may be provided by a single HTTP/1.1 TCP socket, or it might be multiplexed over HTTP/2 or SPDY, or some other protocol.  In all of these cases the abstraction appears the same to the application, as it need not care. The application code typically isn’t required (nor able) to read everything immediately; instead, it is  allowed to read as it processes the data, and this ultimately requires the server to buffer the data in the interim (since other streams should not be blocked). HTTP/2 flow control allows the server to limit the maximum memory usage of this buffering, which would otherwise be significant since HTTP data can be of arbitrary length.

-Jason

On Mar 19, 2015, at 6:54 AM, Bob Briscoe <[hidden email]> wrote:

Patrick,

Thank you for taking the time to read my review carefully. I've been away from mail a few days, which should have allowed time for anything substantive to come from the people with SPDY experience.

We only had the Rob/Greg exchange which merely talked about what some theoretical thing called flow control might be useful for, rather than really addressing my concern that the credit-based protocol provided in h2 is unlikely to be able to provide control if it is needed. Nonetheless, Greg did summarise well the limited usefulness of h2's mechanism.

I should probably have provided a summary of my (long) review, which I try to do below.


I agree that there could be some use for the h2 credit-based protocol at the very start of a stream (in both the C->S and S->C PUSH cases you mentioned). And it might possibly be useful for the cases you mention about unsolicited data, which sound like they occur at the start too. Altho without knowing more about these DoS cases I'm not sure; RST_STREAM might have been sufficient.

However, once a stream has been allowed to open up it's rate, your observation that we mostly see very large windows is what I predicted. It demonstrates that the credit-based protocol does not have sufficient information to be useful to regulate flow. It is effectively being treated like the human appendix - something that no longer serves any purpose but you have to continually put in effort to keep it healthy otherwise it could stop the rest of the system from working.

For this reason, I questioned why flow control has been made mandatory. And I suggested instead that the credit-based flow control in h2 could be
i) mandatory for a data sender to respond to incoming WINDOW_UPDATEs (and therefore a data receiver can gracefully protect itself from DoS by discarding data that exceeds the credit it has previously made available)
ii) optional for a data receiver to emit WINDOW_UPDATEs (i.e. does not even have to implement this part of the code).


Bob

At 21:12 10/03/2015, Patrick McManus wrote:
Hi Bob - I think your comments are appreciated. Its just one of those things where people have dispersed to other things and aren't necessarily in a place to revisit all the ground work again at this stage for a new go round. It was in large part the operational feedback and needs of the google.com team, who has a lot of experience operating spdy at scale, that created the flow control provisions. hopefully those folks will chime in more authoritatively than my musings below:

I'm sure there is quite a bit to learn here - indeed poorly configured use of the window_update mechanism has been (predictably) a source of unintended bottlenecks during both spdy and h2 trials. The spec does try and highlight that there can be dragons here and implementations that don't need the features it can bring should provide essentially infinite credits to steer clear of them.

During the various trials I've seen h2 per stream flow control deployed successfully for a couple of use cases - both of them essentially deal with unsolicited data.

The primary one is essentially a more flexible version of h1's 100-continue. When a client presents a large message body (e.g. a file upload) a multiplexing server needs a way of saying "these are how many buffers I've got available while I figure out where I'm going to sink this incoming data (perhaps to another server I need to connect to)". Presenting this on a per-stream basis allows the server to limit one stream while another (with a distinct sink) can proceed independently.  IMO this value should represent resources available and should be independent of BDP. This is why in practice you see clients with extremely large stream windows - most circumstances just want the data to flow at line rate (as you describe) and aren't trying to invoke flow control. The firefox default window is 256MB per stream - that's not going to slow down the sender nor require frequent window_update generation.

The other use case is when the server pushes resources at a client without them being requested, which is a new feature of h2. This is conceptually similar to the server receiving a POST - suddenly there is a large amount of inbound data that the implementation might not have the resources to store completely. We can't just let TCP flow control take care of the situation because the TCP session is being multiplexed between multiple streams that need to be serviced. In this case the client accepts "some" of the stream based on policy and resource availability and can leave the stream in limbo until an event comes along that tells it to resume the transfer by issuing credits or reject it via cancel.

hth a least a bit.


On Tue, Mar 10, 2015 at 4:31 PM, Bob Briscoe <[hidden email]> wrote:
HTTP/2 folks,

I know extensibility had already been discussed and put to bed, so the WG is entitled to rule out opening healed wounds.

But have points like those I've made about flow control been raised before? Please argue. I may be wrong. Discussion can go on in parallel to the RFC publication process, even tho the process doesn't /require/ you to talk to me.

If I'm right, then implementers are being mandated to write complex flow control code, when it might have little bearing on the performance benefits measured for http/2.

Even if I'm right, and the WG goes ahead anyway, /I/ will understand. My review came in after your deadline.

However, bear in mind that the Webosphere might not be so forgiving. If h2 goes ahead when potential problems have been identified, it could get a bad reputation simply due to the uncertainty, just when you want more people to take it up and try it out. Given you've put in a few person-years of effort, I would have thought you would not want to risk a reputation flop.

I'm trying to help - I just can't go any faster.


Bob


At 14:43 06/03/2015, Bob Briscoe wrote:
HTTP/2 folks,

As I said, consider this as a late review from a clueful but fresh pair of eyes.
My main concerns with the draft are:
* extensibility (previous posting)
* flow control (this posting - apologies for the length - I've tried to explain properly)
* numerous open issues left dangling (see subsequent postings)

===HTTP/2 FLOW CONTROL===

The term 'window' as used throughout is incorrect and highly confusing, in:
* 'flow control window' (44 occurrences),
* 'initial window size' (5),
* or just 'window size' (8)
* and the terms WINDOW_UPDATE and SETTINGS_INITIAL_WINDOW.

The HTTP/2 WINDOW_UPDATE mechanism constrains HTTP/2 to use only credit-based flow control, not window-based. At one point, it actually says it is credit-based (in flow control principle #2 < https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-5.2.1 > ), but otherwise it incorrectly uses the term window.

This is not just an issue of terminology. The more I re-read the flow control sections the more I became convinced that this terminology is not just /confusing/, rather it's evidence of /confusion/. It raises the questions
* "Is HTTP/2 capable of the flow control it says it's capable of?"
* "What type of flow-control protocol ought HTTP/2 to be capable of?"
* "Can the WINDOW_UPDATE frame support the flow-control that HTTP/2 needs?"

To address these questions, it may help if I separate the two different cases HTTP/2 flow control attempts to cover (my own separation, not from the draft):

a) Intermediate buffer control
Here, a stream's flow enters /and/ leaves a buffer (e.g. at the app-layer of an intermediate node).

b) Flow control by the ultimate client app.
Here flow never releases memory (at least not during the life of the connection). The flow is solely consuming more and more memory (e.g. data being rendered into a client app's memory).

==a) Intermediate buffer control==
For this, sliding window-based flow control would be appropriate, because the goal is to keep the e2e pipeline full without wasting buffer.

Let me prove HTTP/2 cannot do window flow control. For window flow control, the sender needs to be able to advance both the leading and trailing edges of the window. In the draft:
* WINDOW_UPDATE frames can only advance the leading edge of a 'window' (and they are constrained to positive values).
* To advance the trailing edge, window flow control would need a continuous stream of acknowledgements back to the sender (like TCP). The draft does not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at the transport layer, so the sending app-layer cannot advance the trailing edge of a 'window'.

So the protocol can only support credit-based flow control. It is incapable of supporting window flow control.

Next, I don't understand how a receiver can set the credit in 'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to answer the question "How much more can I send than I have seen ACK'd?" that would be easy. But because the protocol is restricted to credit, the sender needs the receiver to answer the much harder open-ended question, "How much more can I send?" So the sender needs the receiver to know how many ACKs the sender has seen, but neither of them know that.

The receiver can try, by taking a guess at the bandwidth-delay product, and adjusting the guess up or down, depending on whether its buffer is growing or shrinking. But this only works if the unknown bandwidth-delay product stays constant.

However, BDP will usually be highly variable, as other streams come and go. So, in the time it takes to get a good estimate of the per-stream BDP, it will probably have changed radically, or the stream will most likely have finished anyway. This is why TCP bases flow control on a window, not credit. By complementing window updates with ACK stream info, a TCP sender has sufficient info to control the flow.

The draft is indeed correct when it says:
"   this can lead to suboptimal use of available
   network resources if flow control is enabled without knowledge of the
   bandwidth-delay product (see [RFC7323]).
"
Was this meant to be a veiled criticism of the protocol's own design? A credit-based flow control protocol like that in the draft does not provide sufficient information for either end to estimate the bandwidth-delay product, given it will be varying rapidly.

==b) Control by the ultimate client app==
For this case, I believe neither window nor credit-based flow control is appropriate:
* There is no memory management issue at the client end - even if there's a separate HTTP/2 layer of memory between TCP and the app, it would be pointless to limit the memory used by HTTP/2, because the data is still going to sit in the same user-space memory (or at least about the same amount of memory) when HTTP/2 passes it over for rendering.
* Nonetheless, the receiving client does need to send messages to the sender to supplement stream priorities, by notifying when the state of the receiving application has changed (e.g. if the user's focus switches from one browser tab to another).
* However, credit-based flow control would be very sluggish for such control, because credit cannot be taken back once it has been given (except HTTP/2 allows SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure that hits all streams together).

==Flow control problem summary==

With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.

==Towards a solution?==

I think 'type-a' flow control (for intermediate buffer control) does not need to be at stream-granularity. Indeed, I suspect a proxy could control its app-layer buffering by controlling the receive window of the incoming TCP connection. Has anyone assessed whether this would be sufficient?

I can understand the need for 'type-b' per-stream flow control (by the ultimate client endpoint). Perhaps it would be useful for the receiver to emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream PRIORITY would be sufficient? Either would minimise the response time to a half round trip. Whereas credit flow-control will be much more sluggish (see 'Flow control problem summary').

Either approach would correctly propagate e2e. An intermediate node would naturally tend to prioritise incoming streams that fed into prioritised outgoing streams, so priority updates would tend to propagate from the ultimate receiver, through intermediate nodes, up to the ultimate sender.

==Flow control coverage==

The draft exempts all TCP payload bytes from flow control except HTTP/2 data frames. No rationale is given for this decision. The draft says it's important to manage per-stream memory, then it exempts all the frame types except data, even tho each byte of a non-data frame consumes no less memory than a byte of a data frame.

What message does this put out? "Flow control is not important for one type of bytes with unlimited total size, but flow control is so important that it has to be mandatory for the other type of bytes."

It is certainly critical that WINDOW_UPDATE messages are not covered by flow control, otherwise there would be a real risk of deadlock. It might be that there are dependencies on other frame types that would lead to a dependency loop and deadlock. It would be good to know what the rationale behind these rules was.

==Theory?==

I am concerned that HTTP/2 flow control may have entered new theoretical territory, without suitable proof of safety. The only reassurance we have is one implementation of a flow control algorithm (SPDY), and the anecdotal non-evidence that no-one using SPDY has noticed a deadlock yet (however, is anyone monitoring for deadlocks?).

Whereas SPDY has been an existence proof that an approach like http/2 'works', so far all the flow control algos have been pretty much identical (I think that's true?). I am concerned that the draft takes the InterWeb into uncharted waters, because it allows unconstrained diversity in flow control algos, which is an untested degree of freedom.

The only constraints the draft sets are:
* per-stream flow control is mandatory
* the only protocol message for flow control algos to use is the WINDOW_UPDATE credit message, which cannot be negative
* no constraints on flow control algorithms.
* and all this must work within the outer flow control constraints of TCP.

Some algos might use priority messages to make flow control assumptions. While other algos might associate PRI and WINDOW_UPDATE with different meanings. What confidence do we have that everyone's optimisation algorithms will interoperate? Do we know there will not be certain types of application where deadlock is likely?

"   When using flow
   control, the receiver MUST read from the TCP receive buffer in a
   timely fashion.  Failure to do so could lead to a deadlock when
   critical frames, such as WINDOW_UPDATE, are not read and acted upon.
"
I've been convinced (offlist) that deadlock will not occur as long as the app consumes data 'greedily' from TCP. That has since been articulated in the above normative text. But how sure can we be that every implementer's different interpretations of 'timely' will still prevent deadlock?

Until a good autotuning algorithm for TCP receive window management was developed, good window management code was nearly non-existent. Managing hundreds of interdependent stream buffers is a much harder problem. But implementers are being allowed to just 'Go forth and innovate'. This might work if everyone copies available open source algo(s). But they might not, and they don't have to.

This all seems like 'flying by the seat of the pants'.

==Mandatory Flow Control? ==

"      3. [...] A sender
       MUST respect flow control limits imposed by a receiver."
This ought to be a 'SHOULD' because it is contradicted later - if settings change.

"   6.  Flow control cannot be disabled."
Also effectively contradicted half a page later:
"   Deployments that do not require this capability can advertise a flow
   control window of the maximum size (2^31-1), and by maintaining this
   window by sending a WINDOW_UPDATE frame when any data is received.
   This effectively disables flow control for that receiver."

And contradicted in the definition of half closed (remote):
"  half closed (remote):
      [...] an endpoint is no longer
      obligated to maintain a receiver flow control window.
"

And contradicted in 8.3. The CONNECT Method, which says:
"  Frame types other than DATA
   or stream management frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY)
   MUST NOT be sent on a connected stream, and MUST be treated as a
   stream error (Section 5.4.2) if received.
"

Why is flow control so important that it's mandatory, but so unimportant that you MUST NOT do it when using TLS e2e?

Going back to the earlier quote about using the max window size, it seems perverse for the spec to require endpoints to go through the motions of flow control, even if they arrange for it to affect nothing, but to still require implementation complexity and bandwidth waste with a load of redundant WINDOW_UPDATE frames.

HTTP is used on a wide range of devices, down to the very small and challenged. HTTP/2 might be desirable in such cases, because of the improved efficiency (e.g. header compression), but in many cases the stream model may not be complex enough to need stream flow control.

So why not make flow control optional on the receiving side, but mandatory to implement on the sending side? Then an implementation could have no machinery for tuning window sizes, but it would respond correctly to those set by the other end, which requires much simpler code.

If a receiving implemention chose not to do stream flow control, it could still control flow at the connection (stream 0) level, or at least at the TCP level.


==Inefficiency?==

5.2. Flow Control
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.

==Flow Control Requirements===

I'm not convinced that clear understanding of flow control requirements has driven flow control design decisions.

The draft states various needs for flow-control without giving me a feel of confidence that it has separated out the different cases, and chosen a protocol suitable for each. I tried to go back to the early draft on flow control requirements < http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01 >, and I was not impressed.

I have quoted below the various sentences in the draft that state what flow control is believed to be for. Below that, I have attempted to crystalize out the different concepts, each of which I have tagged within the quotes.

* 2. HTTP/2 Protocol Overview says
  "Flow control and prioritization ensure that it is possible to efficiently use multiplexed streams. [Y]
   Flow control (Section 5.2) helps to ensure that only data that can be used by a receiver is transmitted. [X]"
* 5.2. Flow Control says:
  "Using streams for multiplexing introduces contention over use of the TCP connection [X], resulting in blocked streams [Z]. A flow control scheme ensures that streams on the same connection do not destructively interfere with each other [Z]."
* 5.2.2. Appropriate Use of Flow Control
"  Flow control is defined to protect endpoints that are operating under
   resource constraints.  For example, a proxy needs to share memory
   between many connections, and also might have a slow upstream
   connection and a fast downstream one [Y].  Flow control addresses cases
   where the receiver is unable to process data on one stream, yet wants
   to continue to process other streams in the same connection [X]."
"  Deployments with constrained resources (for example, memory) can
   employ flow control to limit the amount of memory a peer can consume. [Y]

Each requirement has been tagged as follows:
[X] Notification of the receiver's changing utility for each stream
[Y] Prioritisation of streams due to contention over the streaming capacity available to the whole connection.
[Z] Ensuring one stream is not blocked by another.

[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y] sounds more like optimisation across a continuous spectrum.


Regards



Bob

________________________________________________________________
Bob Briscoe,                                                  BT

________________________________________________________________
Bob Briscoe,                                                  BT

________________________________________________________________
Bob Briscoe,                                                  BT


--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Bob Briscoe
Jason,

Yes, I agree this is what we /want/ per-stream flow control to do. My review explained why the flow control in h2 won't be able to do this.

Specifically:

==Flow control problem summary==

With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.


Bob

At 15:08 19/03/2015, Jason Greene wrote:
Hi Bob,

I agree with you that HTTP/2 flow control is not useful for your a) use-case, intermediate buffer control. It is, however, necessary for the b) use case, memory management of an endpoint.

An HTTP server is often divided into a core set of components which manage work scheduling and HTTP protocol aspects, as well as a set of independantly developed  “application” components, that  are often, but not necessarily,  provided by the end-user of the server. There is typically an abstraction between application code and server code, where the application is only ever aware of its stream. The stream may be provided by a single HTTP/1.1 TCP socket, or it might be multiplexed over HTTP/2 or SPDY, or some other protocol.  In all of these cases the abstraction appears the same to the application, as it need not care. The application code typically isn’t required (nor able) to read everything immediately; instead, it is  allowed to read as it processes the data, and this ultimately requires the server to buffer the data in the interim (since other streams should not be blocked). HTTP/2 flow control allows the server to limit the maximum memory usage of this buffering, which would otherwise be significant since HTTP data can be of arbitrary length.

-Jason

On Mar 19, 2015, at 6:54 AM, Bob Briscoe <[hidden email]> wrote:

Patrick,

Thank you for taking the time to read my review carefully. I've been away from mail a few days, which should have allowed time for anything substantive to come from the people with SPDY experience.

We only had the Rob/Greg exchange which merely talked about what some theoretical thing called flow control might be useful for, rather than really addressing my concern that the credit-based protocol provided in h2 is unlikely to be able to provide control if it is needed. Nonetheless, Greg did summarise well the limited usefulness of h2's mechanism.

I should probably have provided a summary of my (long) review, which I try to do below.


I agree that there could be some use for the h2 credit-based protocol at the very start of a stream (in both the C->S and S->C PUSH cases you mentioned). And it might possibly be useful for the cases you mention about unsolicited data, which sound like they occur at the start too. Altho without knowing more about these DoS cases I'm not sure; RST_STREAM might have been sufficient.

However, once a stream has been allowed to open up it's rate, your observation that we mostly see very large windows is what I predicted. It demonstrates that the credit-based protocol does not have sufficient information to be useful to regulate flow. It is effectively being treated like the human appendix - something that no longer serves any purpose but you have to continually put in effort to keep it healthy otherwise it could stop the rest of the system from working.

For this reason, I questioned why flow control has been made mandatory. And I suggested instead that the credit-based flow control in h2 could be
i) mandatory for a data sender to respond to incoming WINDOW_UPDATEs (and therefore a data receiver can gracefully protect itself from DoS by discarding data that exceeds the credit it has previously made available)
ii) optional for a data receiver to emit WINDOW_UPDATEs (i.e. does not even have to implement this part of the code).


Bob

At 21:12 10/03/2015, Patrick McManus wrote:
Hi Bob - I think your comments are appreciated. Its just one of those things where people have dispersed to other things and aren't necessarily in a place to revisit all the ground work again at this stage for a new go round. It was in large part the operational feedback and needs of the google.com team, who has a lot of experience operating spdy at scale, that created the flow control provisions. hopefully those folks will chime in more authoritatively than my musings below:

I'm sure there is quite a bit to learn here - indeed poorly configured use of the window_update mechanism has been (predictably) a source of unintended bottlenecks during both spdy and h2 trials. The spec does try and highlight that there can be dragons here and implementations that don't need the features it can bring should provide essentially infinite credits to steer clear of them.

During the various trials I've seen h2 per stream flow control deployed successfully for a couple of use cases - both of them essentially deal with unsolicited data.

The primary one is essentially a more flexible version of h1's 100-continue. When a client presents a large message body (e.g. a file upload) a multiplexing server needs a way of saying "these are how many buffers I've got available while I figure out where I'm going to sink this incoming data (perhaps to another server I need to connect to)". Presenting this on a per-stream basis allows the server to limit one stream while another (with a distinct sink) can proceed independently.  IMO this value should represent resources available and should be independent of BDP. This is why in practice you see clients with extremely large stream windows - most circumstances just want the data to flow at line rate (as you describe) and aren't trying to invoke flow control. The firefox default window is 256MB per stream - that's not going to slow down the sender nor require frequent window_update generation.

The other use case is when the server pushes resources at a client without them being requested, which is a new feature of h2. This is conceptually similar to the server receiving a POST - suddenly there is a large amount of inbound data that the implementation might not have the resources to store completely. We can't just let TCP flow control take care of the situation because the TCP session is being multiplexed between multiple streams that need to be serviced. In this case the client accepts "some" of the stream based on policy and resource availability and can leave the stream in limbo until an event comes along that tells it to resume the transfer by issuing credits or reject it via cancel.

hth a least a bit.


On Tue, Mar 10, 2015 at 4:31 PM, Bob Briscoe <[hidden email]> wrote:
HTTP/2 folks,
I know extensibility had already been discussed and put to bed, so the WG is entitled to rule out opening healed wounds.
But have points like those I've made about flow control been raised before? Please argue. I may be wrong. Discussion can go on in parallel to the RFC publication process, even tho the process doesn't /require/ you to talk to me.

If I'm right, then implementers are being mandated to write complex flow control code, when it might have little bearing on the performance benefits measured for http/2.
Even if I'm right, and the WG goes ahead anyway, /I/ will understand. My review came in after your deadline.
However, bear in mind that the Webosphere might not be so forgiving. If h2 goes ahead when potential problems have been identified, it could get a bad reputation simply due to the uncertainty, just when you want more people to take it up and try it out. Given you've put in a few person-years of effort, I would have thought you would not want to risk a reputation flop.
I'm trying to help - I just can't go any faster.

Bob

At 14:43 06/03/2015, Bob Briscoe wrote:
HTTP/2 folks,
As I said, consider this as a late review from a clueful but fresh pair of eyes.
My main concerns with the draft are:
* extensibility (previous posting)
* flow control (this posting - apologies for the length - I've tried to explain properly)
* numerous open issues left dangling (see subsequent postings)
===HTTP/2 FLOW CONTROL===
The term 'window' as used throughout is incorrect and highly confusing, in:
* 'flow control window' (44 occurrences),
* 'initial window size' (5),
* or just 'window size' (8)
* and the terms WINDOW_UPDATE and SETTINGS_INITIAL_WINDOW.
The HTTP/2 WINDOW_UPDATE mechanism constrains HTTP/2 to use only credit-based flow control, not window-based. At one point, it actually says it is credit-based (in flow control principle #2 < https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-5.2.1 > ), but otherwise it incorrectly uses the term window.
This is not just an issue of terminology. The more I re-read the flow control sections the more I became convinced that this terminology is not just /confusing/, rather it's evidence of /confusion/. It raises the questions
* "Is HTTP/2 capable of the flow control it says it's capable of?"
* "What type of flow-control protocol ought HTTP/2 to be capable of?"
* "Can the WINDOW_UPDATE frame support the flow-control that HTTP/2 needs?"
To address these questions, it may help if I separate the two different cases HTTP/2 flow control attempts to cover (my own separation, not from the draft):
a) Intermediate buffer control
Here, a stream's flow enters /and/ leaves a buffer (e.g. at the app-layer of an intermediate node).
b) Flow control by the ultimate client app.
Here flow never releases memory (at least not during the life of the connection). The flow is solely consuming more and more memory (e.g. data being rendered into a client app's memory).
==a) Intermediate buffer control==
For this, sliding window-based flow control would be appropriate, because the goal is to keep the e2e pipeline full without wasting buffer.
Let me prove HTTP/2 cannot do window flow control. For window flow control, the sender needs to be able to advance both the leading and trailing edges of the window. In the draft:
* WINDOW_UPDATE frames can only advance the leading edge of a 'window' (and they are constrained to positive values).
* To advance the trailing edge, window flow control would need a continuous stream of acknowledgements back to the sender (like TCP). The draft does not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at the transport layer, so the sending app-layer cannot advance the trailing edge of a 'window'.
So the protocol can only support credit-based flow control. It is incapable of supporting window flow control.
Next, I don't understand how a receiver can set the credit in 'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to answer the question "How much more can I send than I have seen ACK'd?" that would be easy. But because the protocol is restricted to credit, the sender needs the receiver to answer the much harder open-ended question, "How much more can I send?" So the sender needs the receiver to know how many ACKs the sender has seen, but neither of them know that.
The receiver can try, by taking a guess at the bandwidth-delay product, and adjusting the guess up or down, depending on whether its buffer is growing or shrinking. But this only works if the unknown bandwidth-delay product stays constant.
However, BDP will usually be highly variable, as other streams come and go. So, in the time it takes to get a good estimate of the per-stream BDP, it will probably have changed radically, or the stream will most likely have finished anyway. This is why TCP bases flow control on a window, not credit. By complementing window updates with ACK stream info, a TCP sender has sufficient info to control the flow.
The draft is indeed correct when it says:
"   this can lead to suboptimal use of available
   network resources if flow control is enabled without knowledge of the
   bandwidth-delay product (see [RFC7323]).
"
Was this meant to be a veiled criticism of the protocol's own design? A credit-based flow control protocol like that in the draft does not provide sufficient information for either end to estimate the bandwidth-delay product, given it will be varying rapidly.
==b) Control by the ultimate client app==
For this case, I believe neither window nor credit-based flow control is appropriate:
* There is no memory management issue at the client end - even if there's a separate HTTP/2 layer of memory between TCP and the app, it would be pointless to limit the memory used by HTTP/2, because the data is still going to sit in the same user-space memory (or at least about the same amount of memory) when HTTP/2 passes it over for rendering.
* Nonetheless, the receiving client does need to send messages to the sender to supplement stream priorities, by notifying when the state of the receiving application has changed (e.g. if the user's focus switches from one browser tab to another).
* However, credit-based flow control would be very sluggish for such control, because credit cannot be taken back once it has been given (except HTTP/2 allows SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure that hits all streams together).
==Flow control problem summary==
With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.
==Towards a solution?==
I think 'type-a' flow control (for intermediate buffer control) does not need to be at stream-granularity. Indeed, I suspect a proxy could control its app-layer buffering by controlling the receive window of the incoming TCP connection. Has anyone assessed whether this would be sufficient?
I can understand the need for 'type-b' per-stream flow control (by the ultimate client endpoint). Perhaps it would be useful for the receiver to emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream PRIORITY would be sufficient? Either would minimise the response time to a half round trip. Whereas credit flow-control will be much more sluggish (see 'Flow control problem summary').
Either approach would correctly propagate e2e. An intermediate node would naturally tend to prioritise incoming streams that fed into prioritised outgoing streams, so priority updates would tend to propagate from the ultimate receiver, through intermediate nodes, up to the ultimate sender.
==Flow control coverage==
The draft exempts all TCP payload bytes from flow control except HTTP/2 data frames. No rationale is given for this decision. The draft says it's important to manage per-stream memory, then it exempts all the frame types except data, even tho each byte of a non-data frame consumes no less memory than a byte of a data frame.
What message does this put out? "Flow control is not important for one type of bytes with unlimited total size, but flow control is so important that it has to be mandatory for the other type of bytes."
It is certainly critical that WINDOW_UPDATE messages are not covered by flow control, otherwise there would be a real risk of deadlock. It might be that there are dependencies on other frame types that would lead to a dependency loop and deadlock. It would be good to know what the rationale behind these rules was.
==Theory?==
I am concerned that HTTP/2 flow control may have entered new theoretical territory, without suitable proof of safety. The only reassurance we have is one implementation of a flow control algorithm (SPDY), and the anecdotal non-evidence that no-one using SPDY has noticed a deadlock yet (however, is anyone monitoring for deadlocks?).
Whereas SPDY has been an existence proof that an approach like http/2 'works', so far all the flow control algos have been pretty much identical (I think that's true?). I am concerned that the draft takes the InterWeb into uncharted waters, because it allows unconstrained diversity in flow control algos, which is an untested degree of freedom.
The only constraints the draft sets are:
* per-stream flow control is mandatory
* the only protocol message for flow control algos to use is the WINDOW_UPDATE credit message, which cannot be negative
* no constraints on flow control algorithms.
* and all this must work within the outer flow control constraints of TCP.
Some algos might use priority messages to make flow control assumptions. While other algos might associate PRI and WINDOW_UPDATE with different meanings. What confidence do we have that everyone's optimisation algorithms will interoperate? Do we know there will not be certain types of application where deadlock is likely?

"   When using flow
   control, the receiver MUST read from the TCP receive buffer in a
   timely fashion.  Failure to do so could lead to a deadlock when
   critical frames, such as WINDOW_UPDATE, are not read and acted upon.
"
I've been convinced (offlist) that deadlock will not occur as long as the app consumes data 'greedily' from TCP. That has since been articulated in the above normative text. But how sure can we be that every implementer's different interpretations of 'timely' will still prevent deadlock?
Until a good autotuning algorithm for TCP receive window management was developed, good window management code was nearly non-existent. Managing hundreds of interdependent stream buffers is a much harder problem. But implementers are being allowed to just 'Go forth and innovate'. This might work if everyone copies available open source algo(s). But they might not, and they don't have to.
This all seems like 'flying by the seat of the pants'.
==Mandatory Flow Control? ==
"      3. [...] A sender
       MUST respect flow control limits imposed by a receiver."
This ought to be a 'SHOULD' because it is contradicted later - if settings change.
"   6.  Flow control cannot be disabled."
Also effectively contradicted half a page later:
"   Deployments that do not require this capability can advertise a flow
   control window of the maximum size (2^31-1), and by maintaining this
   window by sending a WINDOW_UPDATE frame when any data is received.
   This effectively disables flow control for that receiver."
And contradicted in the definition of half closed (remote):
"  half closed (remote):
      [...] an endpoint is no longer
      obligated to maintain a receiver flow control window.
"
And contradicted in 8.3. The CONNECT Method, which says:
"  Frame types other than DATA
   or stream management frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY)
   MUST NOT be sent on a connected stream, and MUST be treated as a
   stream error (Section 5.4.2) if received.
"
Why is flow control so important that it's mandatory, but so unimportant that you MUST NOT do it when using TLS e2e?
Going back to the earlier quote about using the max window size, it seems perverse for the spec to require endpoints to go through the motions of flow control, even if they arrange for it to affect nothing, but to still require implementation complexity and bandwidth waste with a load of redundant WINDOW_UPDATE frames.
HTTP is used on a wide range of devices, down to the very small and challenged. HTTP/2 might be desirable in such cases, because of the improved efficiency (e.g. header compression), but in many cases the stream model may not be complex enough to need stream flow control.
So why not make flow control optional on the receiving side, but mandatory to implement on the sending side? Then an implementation could have no machinery for tuning window sizes, but it would respond correctly to those set by the other end, which requires much simpler code.
If a receiving implemention chose not to do stream flow control, it could still control flow at the connection (stream 0) level, or at least at the TCP level.

==Inefficiency?==

5.2. Flow Control
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.
==Flow Control Requirements===
I'm not convinced that clear understanding of flow control requirements has driven flow control design decisions.
The draft states various needs for flow-control without giving me a feel of confidence that it has separated out the different cases, and chosen a protocol suitable for each. I tried to go back to the early draft on flow control requirements < http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01 >, and I was not impressed.
I have quoted below the various sentences in the draft that state what flow control is believed to be for. Below that, I have attempted to crystalize out the different concepts, each of which I have tagged within the quotes.
* 2. HTTP/2 Protocol Overview says
  "Flow control and prioritization ensure that it is possible to efficiently use multiplexed streams. [Y]
   Flow control (Section 5.2) helps to ensure that only data that can be used by a receiver is transmitted. [X]"
* 5.2. Flow Control says:
  "Using streams for multiplexing introduces contention over use of the TCP connection [X], resulting in blocked streams [Z]. A flow control scheme ensures that streams on the same connection do not destructively interfere with each other [Z]."
* 5.2.2. Appropriate Use of Flow Control
"  Flow control is defined to protect endpoints that are operating under
   resource constraints.  For example, a proxy needs to share memory
   between many connections, and also might have a slow upstream
   connection and a fast downstream one [Y].  Flow control addresses cases
   where the receiver is unable to process data on one stream, yet wants
   to continue to process other streams in the same connection [X]."
"  Deployments with constrained resources (for example, memory) can
   employ flow control to limit the amount of memory a peer can consume. [Y]
Each requirement has been tagged as follows:
[X] Notification of the receiver's changing utility for each stream
[Y] Prioritisation of streams due to contention over the streaming capacity available to the whole connection.
[Z] Ensuring one stream is not blocked by another.
[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y] sounds more like optimisation across a continuous spectrum.

Regards


Bob
________________________________________________________________
Bob Briscoe,                                                  BT
________________________________________________________________
Bob Briscoe,                                                  BT
________________________________________________________________
Bob Briscoe,                                                  BT

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

________________________________________________________________
Bob Briscoe,                                                  BT

Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Jason Greene
I think thats a good argument for why it’s not suitable for rate-limiting relative to a variable bandwidth product of a single connection (which i took as your use-case a). However, memory limits are typically constant, and an HTTP/2 server can be configured to limit the capacity of the server based on the combination of the “window” size, the number of streams allowed, and the number of connections allowed. Without a hard data credit, the server would have to either HOL block, prevent valid large requests from processing, or run out of resources and start dropping requests. 

On Mar 19, 2015, at 10:38 AM, Bob Briscoe <[hidden email]> wrote:

Jason,

Yes, I agree this is what we /want/ per-stream flow control to do. My review explained why the flow control in h2 won't be able to do this.

Specifically:

==Flow control problem summary==

With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.


Bob

At 15:08 19/03/2015, Jason Greene wrote:
Hi Bob,

I agree with you that HTTP/2 flow control is not useful for your a) use-case, intermediate buffer control. It is, however, necessary for the b) use case, memory management of an endpoint.

An HTTP server is often divided into a core set of components which manage work scheduling and HTTP protocol aspects, as well as a set of independantly developed  “application” components, that  are often, but not necessarily,  provided by the end-user of the server. There is typically an abstraction between application code and server code, where the application is only ever aware of its stream. The stream may be provided by a single HTTP/1.1 TCP socket, or it might be multiplexed over HTTP/2 or SPDY, or some other protocol.  In all of these cases the abstraction appears the same to the application, as it need not care. The application code typically isn’t required (nor able) to read everything immediately; instead, it is  allowed to read as it processes the data, and this ultimately requires the server to buffer the data in the interim (since other streams should not be blocked). HTTP/2 flow control allows the server to limit the maximum memory usage of this buffering, which would otherwise be significant since HTTP data can be of arbitrary length.

-Jason

On Mar 19, 2015, at 6:54 AM, Bob Briscoe <[hidden email]> wrote:

Patrick,

Thank you for taking the time to read my review carefully. I've been away from mail a few days, which should have allowed time for anything substantive to come from the people with SPDY experience.

We only had the Rob/Greg exchange which merely talked about what some theoretical thing called flow control might be useful for, rather than really addressing my concern that the credit-based protocol provided in h2 is unlikely to be able to provide control if it is needed. Nonetheless, Greg did summarise well the limited usefulness of h2's mechanism.

I should probably have provided a summary of my (long) review, which I try to do below.


I agree that there could be some use for the h2 credit-based protocol at the very start of a stream (in both the C->S and S->C PUSH cases you mentioned). And it might possibly be useful for the cases you mention about unsolicited data, which sound like they occur at the start too. Altho without knowing more about these DoS cases I'm not sure; RST_STREAM might have been sufficient.

However, once a stream has been allowed to open up it's rate, your observation that we mostly see very large windows is what I predicted. It demonstrates that the credit-based protocol does not have sufficient information to be useful to regulate flow. It is effectively being treated like the human appendix - something that no longer serves any purpose but you have to continually put in effort to keep it healthy otherwise it could stop the rest of the system from working.

For this reason, I questioned why flow control has been made mandatory. And I suggested instead that the credit-based flow control in h2 could be
i) mandatory for a data sender to respond to incoming WINDOW_UPDATEs (and therefore a data receiver can gracefully protect itself from DoS by discarding data that exceeds the credit it has previously made available)
ii) optional for a data receiver to emit WINDOW_UPDATEs (i.e. does not even have to implement this part of the code).


Bob

At 21:12 10/03/2015, Patrick McManus wrote:
Hi Bob - I think your comments are appreciated. Its just one of those things where people have dispersed to other things and aren't necessarily in a place to revisit all the ground work again at this stage for a new go round. It was in large part the operational feedback and needs of the google.com team, who has a lot of experience operating spdy at scale, that created the flow control provisions. hopefully those folks will chime in more authoritatively than my musings below:

I'm sure there is quite a bit to learn here - indeed poorly configured use of the window_update mechanism has been (predictably) a source of unintended bottlenecks during both spdy and h2 trials. The spec does try and highlight that there can be dragons here and implementations that don't need the features it can bring should provide essentially infinite credits to steer clear of them.

During the various trials I've seen h2 per stream flow control deployed successfully for a couple of use cases - both of them essentially deal with unsolicited data.

The primary one is essentially a more flexible version of h1's 100-continue. When a client presents a large message body (e.g. a file upload) a multiplexing server needs a way of saying "these are how many buffers I've got available while I figure out where I'm going to sink this incoming data (perhaps to another server I need to connect to)". Presenting this on a per-stream basis allows the server to limit one stream while another (with a distinct sink) can proceed independently.  IMO this value should represent resources available and should be independent of BDP. This is why in practice you see clients with extremely large stream windows - most circumstances just want the data to flow at line rate (as you describe) and aren't trying to invoke flow control. The firefox default window is 256MB per stream - that's not going to slow down the sender nor require frequent window_update generation.

The other use case is when the server pushes resources at a client without them being requested, which is a new feature of h2. This is conceptually similar to the server receiving a POST - suddenly there is a large amount of inbound data that the implementation might not have the resources to store completely. We can't just let TCP flow control take care of the situation because the TCP session is being multiplexed between multiple streams that need to be serviced. In this case the client accepts "some" of the stream based on policy and resource availability and can leave the stream in limbo until an event comes along that tells it to resume the transfer by issuing credits or reject it via cancel.

hth a least a bit.


On Tue, Mar 10, 2015 at 4:31 PM, Bob Briscoe <[hidden email]> wrote:
HTTP/2 folks,
I know extensibility had already been discussed and put to bed, so the WG is entitled to rule out opening healed wounds.
But have points like those I've made about flow control been raised before? Please argue. I may be wrong. Discussion can go on in parallel to the RFC publication process, even tho the process doesn't /require/ you to talk to me.

If I'm right, then implementers are being mandated to write complex flow control code, when it might have little bearing on the performance benefits measured for http/2.
Even if I'm right, and the WG goes ahead anyway, /I/ will understand. My review came in after your deadline.
However, bear in mind that the Webosphere might not be so forgiving. If h2 goes ahead when potential problems have been identified, it could get a bad reputation simply due to the uncertainty, just when you want more people to take it up and try it out. Given you've put in a few person-years of effort, I would have thought you would not want to risk a reputation flop.
I'm trying to help - I just can't go any faster.

Bob

At 14:43 06/03/2015, Bob Briscoe wrote:
HTTP/2 folks,
As I said, consider this as a late review from a clueful but fresh pair of eyes.
My main concerns with the draft are:
* extensibility (previous posting)
* flow control (this posting - apologies for the length - I've tried to explain properly)
* numerous open issues left dangling (see subsequent postings)
===HTTP/2 FLOW CONTROL===
The term 'window' as used throughout is incorrect and highly confusing, in:
* 'flow control window' (44 occurrences),
* 'initial window size' (5),
* or just 'window size' (8)
* and the terms WINDOW_UPDATE and SETTINGS_INITIAL_WINDOW.
The HTTP/2 WINDOW_UPDATE mechanism constrains HTTP/2 to use only credit-based flow control, not window-based. At one point, it actually says it is credit-based (in flow control principle #2 < https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-5.2.1 > ), but otherwise it incorrectly uses the term window.
This is not just an issue of terminology. The more I re-read the flow control sections the more I became convinced that this terminology is not just /confusing/, rather it's evidence of /confusion/. It raises the questions
* "Is HTTP/2 capable of the flow control it says it's capable of?"
* "What type of flow-control protocol ought HTTP/2 to be capable of?"
* "Can the WINDOW_UPDATE frame support the flow-control that HTTP/2 needs?"
To address these questions, it may help if I separate the two different cases HTTP/2 flow control attempts to cover (my own separation, not from the draft):
a) Intermediate buffer control
Here, a stream's flow enters /and/ leaves a buffer (e.g. at the app-layer of an intermediate node).
b) Flow control by the ultimate client app.
Here flow never releases memory (at least not during the life of the connection). The flow is solely consuming more and more memory (e.g. data being rendered into a client app's memory).
==a) Intermediate buffer control==
For this, sliding window-based flow control would be appropriate, because the goal is to keep the e2e pipeline full without wasting buffer.
Let me prove HTTP/2 cannot do window flow control. For window flow control, the sender needs to be able to advance both the leading and trailing edges of the window. In the draft:
* WINDOW_UPDATE frames can only advance the leading edge of a 'window' (and they are constrained to positive values).
* To advance the trailing edge, window flow control would need a continuous stream of acknowledgements back to the sender (like TCP). The draft does not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at the transport layer, so the sending app-layer cannot advance the trailing edge of a 'window'.
So the protocol can only support credit-based flow control. It is incapable of supporting window flow control.
Next, I don't understand how a receiver can set the credit in 'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to answer the question "How much more can I send than I have seen ACK'd?" that would be easy. But because the protocol is restricted to credit, the sender needs the receiver to answer the much harder open-ended question, "How much more can I send?" So the sender needs the receiver to know how many ACKs the sender has seen, but neither of them know that.
The receiver can try, by taking a guess at the bandwidth-delay product, and adjusting the guess up or down, depending on whether its buffer is growing or shrinking. But this only works if the unknown bandwidth-delay product stays constant.
However, BDP will usually be highly variable, as other streams come and go. So, in the time it takes to get a good estimate of the per-stream BDP, it will probably have changed radically, or the stream will most likely have finished anyway. This is why TCP bases flow control on a window, not credit. By complementing window updates with ACK stream info, a TCP sender has sufficient info to control the flow.
The draft is indeed correct when it says:
"   this can lead to suboptimal use of available
   network resources if flow control is enabled without knowledge of the
   bandwidth-delay product (see [RFC7323]).
"
Was this meant to be a veiled criticism of the protocol's own design? A credit-based flow control protocol like that in the draft does not provide sufficient information for either end to estimate the bandwidth-delay product, given it will be varying rapidly.
==b) Control by the ultimate client app==
For this case, I believe neither window nor credit-based flow control is appropriate:
* There is no memory management issue at the client end - even if there's a separate HTTP/2 layer of memory between TCP and the app, it would be pointless to limit the memory used by HTTP/2, because the data is still going to sit in the same user-space memory (or at least about the same amount of memory) when HTTP/2 passes it over for rendering.
* Nonetheless, the receiving client does need to send messages to the sender to supplement stream priorities, by notifying when the state of the receiving application has changed (e.g. if the user's focus switches from one browser tab to another).
* However, credit-based flow control would be very sluggish for such control, because credit cannot be taken back once it has been given (except HTTP/2 allows SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure that hits all streams together).
==Flow control problem summary==
With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.
==Towards a solution?==
I think 'type-a' flow control (for intermediate buffer control) does not need to be at stream-granularity. Indeed, I suspect a proxy could control its app-layer buffering by controlling the receive window of the incoming TCP connection. Has anyone assessed whether this would be sufficient?
I can understand the need for 'type-b' per-stream flow control (by the ultimate client endpoint). Perhaps it would be useful for the receiver to emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream PRIORITY would be sufficient? Either would minimise the response time to a half round trip. Whereas credit flow-control will be much more sluggish (see 'Flow control problem summary').
Either approach would correctly propagate e2e. An intermediate node would naturally tend to prioritise incoming streams that fed into prioritised outgoing streams, so priority updates would tend to propagate from the ultimate receiver, through intermediate nodes, up to the ultimate sender.
==Flow control coverage==
The draft exempts all TCP payload bytes from flow control except HTTP/2 data frames. No rationale is given for this decision. The draft says it's important to manage per-stream memory, then it exempts all the frame types except data, even tho each byte of a non-data frame consumes no less memory than a byte of a data frame.
What message does this put out? "Flow control is not important for one type of bytes with unlimited total size, but flow control is so important that it has to be mandatory for the other type of bytes."
It is certainly critical that WINDOW_UPDATE messages are not covered by flow control, otherwise there would be a real risk of deadlock. It might be that there are dependencies on other frame types that would lead to a dependency loop and deadlock. It would be good to know what the rationale behind these rules was.
==Theory?==
I am concerned that HTTP/2 flow control may have entered new theoretical territory, without suitable proof of safety. The only reassurance we have is one implementation of a flow control algorithm (SPDY), and the anecdotal non-evidence that no-one using SPDY has noticed a deadlock yet (however, is anyone monitoring for deadlocks?).
Whereas SPDY has been an existence proof that an approach like http/2 'works', so far all the flow control algos have been pretty much identical (I think that's true?). I am concerned that the draft takes the InterWeb into uncharted waters, because it allows unconstrained diversity in flow control algos, which is an untested degree of freedom.
The only constraints the draft sets are:
* per-stream flow control is mandatory
* the only protocol message for flow control algos to use is the WINDOW_UPDATE credit message, which cannot be negative
* no constraints on flow control algorithms.
* and all this must work within the outer flow control constraints of TCP.
Some algos might use priority messages to make flow control assumptions. While other algos might associate PRI and WINDOW_UPDATE with different meanings. What confidence do we have that everyone's optimisation algorithms will interoperate? Do we know there will not be certain types of application where deadlock is likely?

"   When using flow
   control, the receiver MUST read from the TCP receive buffer in a
   timely fashion.  Failure to do so could lead to a deadlock when
   critical frames, such as WINDOW_UPDATE, are not read and acted upon.
"
I've been convinced (offlist) that deadlock will not occur as long as the app consumes data 'greedily' from TCP. That has since been articulated in the above normative text. But how sure can we be that every implementer's different interpretations of 'timely' will still prevent deadlock?
Until a good autotuning algorithm for TCP receive window management was developed, good window management code was nearly non-existent. Managing hundreds of interdependent stream buffers is a much harder problem. But implementers are being allowed to just 'Go forth and innovate'. This might work if everyone copies available open source algo(s). But they might not, and they don't have to.
This all seems like 'flying by the seat of the pants'.
==Mandatory Flow Control? ==
"      3. [...] A sender
       MUST respect flow control limits imposed by a receiver."
This ought to be a 'SHOULD' because it is contradicted later - if settings change.
"   6.  Flow control cannot be disabled."
Also effectively contradicted half a page later:
"   Deployments that do not require this capability can advertise a flow
   control window of the maximum size (2^31-1), and by maintaining this
   window by sending a WINDOW_UPDATE frame when any data is received.
   This effectively disables flow control for that receiver."
And contradicted in the definition of half closed (remote):
"  half closed (remote):
      [...] an endpoint is no longer
      obligated to maintain a receiver flow control window.
"
And contradicted in 8.3. The CONNECT Method, which says:
"  Frame types other than DATA
   or stream management frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY)
   MUST NOT be sent on a connected stream, and MUST be treated as a
   stream error (Section 5.4.2) if received.
"
Why is flow control so important that it's mandatory, but so unimportant that you MUST NOT do it when using TLS e2e?
Going back to the earlier quote about using the max window size, it seems perverse for the spec to require endpoints to go through the motions of flow control, even if they arrange for it to affect nothing, but to still require implementation complexity and bandwidth waste with a load of redundant WINDOW_UPDATE frames.
HTTP is used on a wide range of devices, down to the very small and challenged. HTTP/2 might be desirable in such cases, because of the improved efficiency (e.g. header compression), but in many cases the stream model may not be complex enough to need stream flow control.
So why not make flow control optional on the receiving side, but mandatory to implement on the sending side? Then an implementation could have no machinery for tuning window sizes, but it would respond correctly to those set by the other end, which requires much simpler code.
If a receiving implemention chose not to do stream flow control, it could still control flow at the connection (stream 0) level, or at least at the TCP level.

==Inefficiency?==

5.2. Flow Control
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.
==Flow Control Requirements===
I'm not convinced that clear understanding of flow control requirements has driven flow control design decisions.
The draft states various needs for flow-control without giving me a feel of confidence that it has separated out the different cases, and chosen a protocol suitable for each. I tried to go back to the early draft on flow control requirements < http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01 >, and I was not impressed.
I have quoted below the various sentences in the draft that state what flow control is believed to be for. Below that, I have attempted to crystalize out the different concepts, each of which I have tagged within the quotes.
* 2. HTTP/2 Protocol Overview says
  "Flow control and prioritization ensure that it is possible to efficiently use multiplexed streams. [Y]
   Flow control (Section 5.2) helps to ensure that only data that can be used by a receiver is transmitted. [X]"
* 5.2. Flow Control says:
  "Using streams for multiplexing introduces contention over use of the TCP connection [X], resulting in blocked streams [Z]. A flow control scheme ensures that streams on the same connection do not destructively interfere with each other [Z]."
* 5.2.2. Appropriate Use of Flow Control
"  Flow control is defined to protect endpoints that are operating under
   resource constraints.  For example, a proxy needs to share memory
   between many connections, and also might have a slow upstream
   connection and a fast downstream one [Y].  Flow control addresses cases
   where the receiver is unable to process data on one stream, yet wants
   to continue to process other streams in the same connection [X]."
"  Deployments with constrained resources (for example, memory) can
   employ flow control to limit the amount of memory a peer can consume. [Y]
Each requirement has been tagged as follows:
[X] Notification of the receiver's changing utility for each stream
[Y] Prioritisation of streams due to contention over the streaming capacity available to the whole connection.
[Z] Ensuring one stream is not blocked by another.
[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y] sounds more like optimisation across a continuous spectrum.

Regards


Bob
________________________________________________________________
Bob Briscoe,                                                  BT
________________________________________________________________
Bob Briscoe,                                                  BT
________________________________________________________________
Bob Briscoe,                                                  BT

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

________________________________________________________________
Bob Briscoe,                                                  BT


--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Roberto Peon-2
In reply to this post by Bob Briscoe

H2 does both connection-level and stream-level flow control. Both are necessary to prevent infinite buffering if one assumes, as intended, that the max number of streams should be very large.

The per-stream control is necessary

On Mar 19, 2015 9:01 AM, "Jason Greene" <[hidden email]> wrote:
I think thats a good argument for why it’s not suitable for rate-limiting relative to a variable bandwidth product of a single connection (which i took as your use-case a). However, memory limits are typically constant, and an HTTP/2 server can be configured to limit the capacity of the server based on the combination of the “window” size, the number of streams allowed, and the number of connections allowed. Without a hard data credit, the server would have to either HOL block, prevent valid large requests from processing, or run out of resources and start dropping requests. 

On Mar 19, 2015, at 10:38 AM, Bob Briscoe <[hidden email]> wrote:

Jason,

Yes, I agree this is what we /want/ per-stream flow control to do. My review explained why the flow control in h2 won't be able to do this.

Specifically:

==Flow control problem summary==

With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.


Bob

At 15:08 19/03/2015, Jason Greene wrote:
Hi Bob,

I agree with you that HTTP/2 flow control is not useful for your a) use-case, intermediate buffer control. It is, however, necessary for the b) use case, memory management of an endpoint.

An HTTP server is often divided into a core set of components which manage work scheduling and HTTP protocol aspects, as well as a set of independantly developed  “application” components, that  are often, but not necessarily,  provided by the end-user of the server. There is typically an abstraction between application code and server code, where the application is only ever aware of its stream. The stream may be provided by a single HTTP/1.1 TCP socket, or it might be multiplexed over HTTP/2 or SPDY, or some other protocol.  In all of these cases the abstraction appears the same to the application, as it need not care. The application code typically isn’t required (nor able) to read everything immediately; instead, it is  allowed to read as it processes the data, and this ultimately requires the server to buffer the data in the interim (since other streams should not be blocked). HTTP/2 flow control allows the server to limit the maximum memory usage of this buffering, which would otherwise be significant since HTTP data can be of arbitrary length.

-Jason

On Mar 19, 2015, at 6:54 AM, Bob Briscoe <[hidden email]> wrote:

Patrick,

Thank you for taking the time to read my review carefully. I've been away from mail a few days, which should have allowed time for anything substantive to come from the people with SPDY experience.

We only had the Rob/Greg exchange which merely talked about what some theoretical thing called flow control might be useful for, rather than really addressing my concern that the credit-based protocol provided in h2 is unlikely to be able to provide control if it is needed. Nonetheless, Greg did summarise well the limited usefulness of h2's mechanism.

I should probably have provided a summary of my (long) review, which I try to do below.


I agree that there could be some use for the h2 credit-based protocol at the very start of a stream (in both the C->S and S->C PUSH cases you mentioned). And it might possibly be useful for the cases you mention about unsolicited data, which sound like they occur at the start too. Altho without knowing more about these DoS cases I'm not sure; RST_STREAM might have been sufficient.

However, once a stream has been allowed to open up it's rate, your observation that we mostly see very large windows is what I predicted. It demonstrates that the credit-based protocol does not have sufficient information to be useful to regulate flow. It is effectively being treated like the human appendix - something that no longer serves any purpose but you have to continually put in effort to keep it healthy otherwise it could stop the rest of the system from working.

For this reason, I questioned why flow control has been made mandatory. And I suggested instead that the credit-based flow control in h2 could be
i) mandatory for a data sender to respond to incoming WINDOW_UPDATEs (and therefore a data receiver can gracefully protect itself from DoS by discarding data that exceeds the credit it has previously made available)
ii) optional for a data receiver to emit WINDOW_UPDATEs (i.e. does not even have to implement this part of the code).


Bob

At 21:12 10/03/2015, Patrick McManus wrote:
Hi Bob - I think your comments are appreciated. Its just one of those things where people have dispersed to other things and aren't necessarily in a place to revisit all the ground work again at this stage for a new go round. It was in large part the operational feedback and needs of the google.com team, who has a lot of experience operating spdy at scale, that created the flow control provisions. hopefully those folks will chime in more authoritatively than my musings below:

I'm sure there is quite a bit to learn here - indeed poorly configured use of the window_update mechanism has been (predictably) a source of unintended bottlenecks during both spdy and h2 trials. The spec does try and highlight that there can be dragons here and implementations that don't need the features it can bring should provide essentially infinite credits to steer clear of them.

During the various trials I've seen h2 per stream flow control deployed successfully for a couple of use cases - both of them essentially deal with unsolicited data.

The primary one is essentially a more flexible version of h1's 100-continue. When a client presents a large message body (e.g. a file upload) a multiplexing server needs a way of saying "these are how many buffers I've got available while I figure out where I'm going to sink this incoming data (perhaps to another server I need to connect to)". Presenting this on a per-stream basis allows the server to limit one stream while another (with a distinct sink) can proceed independently.  IMO this value should represent resources available and should be independent of BDP. This is why in practice you see clients with extremely large stream windows - most circumstances just want the data to flow at line rate (as you describe) and aren't trying to invoke flow control. The firefox default window is 256MB per stream - that's not going to slow down the sender nor require frequent window_update generation.

The other use case is when the server pushes resources at a client without them being requested, which is a new feature of h2. This is conceptually similar to the server receiving a POST - suddenly there is a large amount of inbound data that the implementation might not have the resources to store completely. We can't just let TCP flow control take care of the situation because the TCP session is being multiplexed between multiple streams that need to be serviced. In this case the client accepts "some" of the stream based on policy and resource availability and can leave the stream in limbo until an event comes along that tells it to resume the transfer by issuing credits or reject it via cancel.

hth a least a bit.


On Tue, Mar 10, 2015 at 4:31 PM, Bob Briscoe <[hidden email]> wrote:
HTTP/2 folks,
I know extensibility had already been discussed and put to bed, so the WG is entitled to rule out opening healed wounds.
But have points like those I've made about flow control been raised before? Please argue. I may be wrong. Discussion can go on in parallel to the RFC publication process, even tho the process doesn't /require/ you to talk to me.

If I'm right, then implementers are being mandated to write complex flow control code, when it might have little bearing on the performance benefits measured for http/2.
Even if I'm right, and the WG goes ahead anyway, /I/ will understand. My review came in after your deadline.
However, bear in mind that the Webosphere might not be so forgiving. If h2 goes ahead when potential problems have been identified, it could get a bad reputation simply due to the uncertainty, just when you want more people to take it up and try it out. Given you've put in a few person-years of effort, I would have thought you would not want to risk a reputation flop.
I'm trying to help - I just can't go any faster.

Bob

At 14:43 06/03/2015, Bob Briscoe wrote:
HTTP/2 folks,
As I said, consider this as a late review from a clueful but fresh pair of eyes.
My main concerns with the draft are:
* extensibility (previous posting)
* flow control (this posting - apologies for the length - I've tried to explain properly)
* numerous open issues left dangling (see subsequent postings)
===HTTP/2 FLOW CONTROL===
The term 'window' as used throughout is incorrect and highly confusing, in:
* 'flow control window' (44 occurrences),
* 'initial window size' (5),
* or just 'window size' (8)
* and the terms WINDOW_UPDATE and SETTINGS_INITIAL_WINDOW.
The HTTP/2 WINDOW_UPDATE mechanism constrains HTTP/2 to use only credit-based flow control, not window-based. At one point, it actually says it is credit-based (in flow control principle #2 < https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-5.2.1 > ), but otherwise it incorrectly uses the term window.
This is not just an issue of terminology. The more I re-read the flow control sections the more I became convinced that this terminology is not just /confusing/, rather it's evidence of /confusion/. It raises the questions
* "Is HTTP/2 capable of the flow control it says it's capable of?"
* "What type of flow-control protocol ought HTTP/2 to be capable of?"
* "Can the WINDOW_UPDATE frame support the flow-control that HTTP/2 needs?"
To address these questions, it may help if I separate the two different cases HTTP/2 flow control attempts to cover (my own separation, not from the draft):
a) Intermediate buffer control
Here, a stream's flow enters /and/ leaves a buffer (e.g. at the app-layer of an intermediate node).
b) Flow control by the ultimate client app.
Here flow never releases memory (at least not during the life of the connection). The flow is solely consuming more and more memory (e.g. data being rendered into a client app's memory).
==a) Intermediate buffer control==
For this, sliding window-based flow control would be appropriate, because the goal is to keep the e2e pipeline full without wasting buffer.
Let me prove HTTP/2 cannot do window flow control. For window flow control, the sender needs to be able to advance both the leading and trailing edges of the window. In the draft:
* WINDOW_UPDATE frames can only advance the leading edge of a 'window' (and they are constrained to positive values).
* To advance the trailing edge, window flow control would need a continuous stream of acknowledgements back to the sender (like TCP). The draft does not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at the transport layer, so the sending app-layer cannot advance the trailing edge of a 'window'.
So the protocol can only support credit-based flow control. It is incapable of supporting window flow control.
Next, I don't understand how a receiver can set the credit in 'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to answer the question "How much more can I send than I have seen ACK'd?" that would be easy. But because the protocol is restricted to credit, the sender needs the receiver to answer the much harder open-ended question, "How much more can I send?" So the sender needs the receiver to know how many ACKs the sender has seen, but neither of them know that.
The receiver can try, by taking a guess at the bandwidth-delay product, and adjusting the guess up or down, depending on whether its buffer is growing or shrinking. But this only works if the unknown bandwidth-delay product stays constant.
However, BDP will usually be highly variable, as other streams come and go. So, in the time it takes to get a good estimate of the per-stream BDP, it will probably have changed radically, or the stream will most likely have finished anyway. This is why TCP bases flow control on a window, not credit. By complementing window updates with ACK stream info, a TCP sender has sufficient info to control the flow.
The draft is indeed correct when it says:
"   this can lead to suboptimal use of available
   network resources if flow control is enabled without knowledge of the
   bandwidth-delay product (see [RFC7323]).
"
Was this meant to be a veiled criticism of the protocol's own design? A credit-based flow control protocol like that in the draft does not provide sufficient information for either end to estimate the bandwidth-delay product, given it will be varying rapidly.
==b) Control by the ultimate client app==
For this case, I believe neither window nor credit-based flow control is appropriate:
* There is no memory management issue at the client end - even if there's a separate HTTP/2 layer of memory between TCP and the app, it would be pointless to limit the memory used by HTTP/2, because the data is still going to sit in the same user-space memory (or at least about the same amount of memory) when HTTP/2 passes it over for rendering.
* Nonetheless, the receiving client does need to send messages to the sender to supplement stream priorities, by notifying when the state of the receiving application has changed (e.g. if the user's focus switches from one browser tab to another).
* However, credit-based flow control would be very sluggish for such control, because credit cannot be taken back once it has been given (except HTTP/2 allows SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure that hits all streams together).
==Flow control problem summary==
With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.
==Towards a solution?==
I think 'type-a' flow control (for intermediate buffer control) does not need to be at stream-granularity. Indeed, I suspect a proxy could control its app-layer buffering by controlling the receive window of the incoming TCP connection. Has anyone assessed whether this would be sufficient?
I can understand the need for 'type-b' per-stream flow control (by the ultimate client endpoint). Perhaps it would be useful for the receiver to emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream PRIORITY would be sufficient? Either would minimise the response time to a half round trip. Whereas credit flow-control will be much more sluggish (see 'Flow control problem summary').
Either approach would correctly propagate e2e. An intermediate node would naturally tend to prioritise incoming streams that fed into prioritised outgoing streams, so priority updates would tend to propagate from the ultimate receiver, through intermediate nodes, up to the ultimate sender.
==Flow control coverage==
The draft exempts all TCP payload bytes from flow control except HTTP/2 data frames. No rationale is given for this decision. The draft says it's important to manage per-stream memory, then it exempts all the frame types except data, even tho each byte of a non-data frame consumes no less memory than a byte of a data frame.
What message does this put out? "Flow control is not important for one type of bytes with unlimited total size, but flow control is so important that it has to be mandatory for the other type of bytes."
It is certainly critical that WINDOW_UPDATE messages are not covered by flow control, otherwise there would be a real risk of deadlock. It might be that there are dependencies on other frame types that would lead to a dependency loop and deadlock. It would be good to know what the rationale behind these rules was.
==Theory?==
I am concerned that HTTP/2 flow control may have entered new theoretical territory, without suitable proof of safety. The only reassurance we have is one implementation of a flow control algorithm (SPDY), and the anecdotal non-evidence that no-one using SPDY has noticed a deadlock yet (however, is anyone monitoring for deadlocks?).
Whereas SPDY has been an existence proof that an approach like http/2 'works', so far all the flow control algos have been pretty much identical (I think that's true?). I am concerned that the draft takes the InterWeb into uncharted waters, because it allows unconstrained diversity in flow control algos, which is an untested degree of freedom.
The only constraints the draft sets are:
* per-stream flow control is mandatory
* the only protocol message for flow control algos to use is the WINDOW_UPDATE credit message, which cannot be negative
* no constraints on flow control algorithms.
* and all this must work within the outer flow control constraints of TCP.
Some algos might use priority messages to make flow control assumptions. While other algos might associate PRI and WINDOW_UPDATE with different meanings. What confidence do we have that everyone's optimisation algorithms will interoperate? Do we know there will not be certain types of application where deadlock is likely?

"   When using flow
   control, the receiver MUST read from the TCP receive buffer in a
   timely fashion.  Failure to do so could lead to a deadlock when
   critical frames, such as WINDOW_UPDATE, are not read and acted upon.
"
I've been convinced (offlist) that deadlock will not occur as long as the app consumes data 'greedily' from TCP. That has since been articulated in the above normative text. But how sure can we be that every implementer's different interpretations of 'timely' will still prevent deadlock?
Until a good autotuning algorithm for TCP receive window management was developed, good window management code was nearly non-existent. Managing hundreds of interdependent stream buffers is a much harder problem. But implementers are being allowed to just 'Go forth and innovate'. This might work if everyone copies available open source algo(s). But they might not, and they don't have to.
This all seems like 'flying by the seat of the pants'.
==Mandatory Flow Control? ==
"      3. [...] A sender
       MUST respect flow control limits imposed by a receiver."
This ought to be a 'SHOULD' because it is contradicted later - if settings change.
"   6.  Flow control cannot be disabled."
Also effectively contradicted half a page later:
"   Deployments that do not require this capability can advertise a flow
   control window of the maximum size (2^31-1), and by maintaining this
   window by sending a WINDOW_UPDATE frame when any data is received.
   This effectively disables flow control for that receiver."
And contradicted in the definition of half closed (remote):
"  half closed (remote):
      [...] an endpoint is no longer
      obligated to maintain a receiver flow control window.
"
And contradicted in 8.3. The CONNECT Method, which says:
"  Frame types other than DATA
   or stream management frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY)
   MUST NOT be sent on a connected stream, and MUST be treated as a
   stream error (Section 5.4.2) if received.
"
Why is flow control so important that it's mandatory, but so unimportant that you MUST NOT do it when using TLS e2e?
Going back to the earlier quote about using the max window size, it seems perverse for the spec to require endpoints to go through the motions of flow control, even if they arrange for it to affect nothing, but to still require implementation complexity and bandwidth waste with a load of redundant WINDOW_UPDATE frames.
HTTP is used on a wide range of devices, down to the very small and challenged. HTTP/2 might be desirable in such cases, because of the improved efficiency (e.g. header compression), but in many cases the stream model may not be complex enough to need stream flow control.
So why not make flow control optional on the receiving side, but mandatory to implement on the sending side? Then an implementation could have no machinery for tuning window sizes, but it would respond correctly to those set by the other end, which requires much simpler code.
If a receiving implemention chose not to do stream flow control, it could still control flow at the connection (stream 0) level, or at least at the TCP level.

==Inefficiency?==

5.2. Flow Control
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.
==Flow Control Requirements===
I'm not convinced that clear understanding of flow control requirements has driven flow control design decisions.
The draft states various needs for flow-control without giving me a feel of confidence that it has separated out the different cases, and chosen a protocol suitable for each. I tried to go back to the early draft on flow control requirements < http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01 >, and I was not impressed.
I have quoted below the various sentences in the draft that state what flow control is believed to be for. Below that, I have attempted to crystalize out the different concepts, each of which I have tagged within the quotes.
* 2. HTTP/2 Protocol Overview says
  "Flow control and prioritization ensure that it is possible to efficiently use multiplexed streams. [Y]
   Flow control (Section 5.2) helps to ensure that only data that can be used by a receiver is transmitted. [X]"
* 5.2. Flow Control says:
  "Using streams for multiplexing introduces contention over use of the TCP connection [X], resulting in blocked streams [Z]. A flow control scheme ensures that streams on the same connection do not destructively interfere with each other [Z]."
* 5.2.2. Appropriate Use of Flow Control
"  Flow control is defined to protect endpoints that are operating under
   resource constraints.  For example, a proxy needs to share memory
   between many connections, and also might have a slow upstream
   connection and a fast downstream one [Y].  Flow control addresses cases
   where the receiver is unable to process data on one stream, yet wants
   to continue to process other streams in the same connection [X]."
"  Deployments with constrained resources (for example, memory) can
   employ flow control to limit the amount of memory a peer can consume. [Y]
Each requirement has been tagged as follows:
[X] Notification of the receiver's changing utility for each stream
[Y] Prioritisation of streams due to contention over the streaming capacity available to the whole connection.
[Z] Ensuring one stream is not blocked by another.
[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y] sounds more like optimisation across a continuous spectrum.

Regards


Bob
________________________________________________________________
Bob Briscoe,                                                  BT
________________________________________________________________
Bob Briscoe,                                                  BT
________________________________________________________________
Bob Briscoe,                                                  BT

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

________________________________________________________________
Bob Briscoe,                                                  BT


--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Roberto Peon-2

Disregard-- this was an incomplete draft that got sent instead of discarded.

Suffice it to say that the flow control is good enough to prevent oom on servers and intermediaries. It is otherwise imperfect, though simple.

-=R

On Mar 19, 2015 3:26 PM, "Roberto Peon" <[hidden email]> wrote:

H2 does both connection-level and stream-level flow control. Both are necessary to prevent infinite buffering if one assumes, as intended, that the max number of streams should be very large.

The per-stream control is necessary

On Mar 19, 2015 9:01 AM, "Jason Greene" <[hidden email]> wrote:
I think thats a good argument for why it’s not suitable for rate-limiting relative to a variable bandwidth product of a single connection (which i took as your use-case a). However, memory limits are typically constant, and an HTTP/2 server can be configured to limit the capacity of the server based on the combination of the “window” size, the number of streams allowed, and the number of connections allowed. Without a hard data credit, the server would have to either HOL block, prevent valid large requests from processing, or run out of resources and start dropping requests. 

On Mar 19, 2015, at 10:38 AM, Bob Briscoe <[hidden email]> wrote:

Jason,

Yes, I agree this is what we /want/ per-stream flow control to do. My review explained why the flow control in h2 won't be able to do this.

Specifically:

==Flow control problem summary==

With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.


Bob

At 15:08 19/03/2015, Jason Greene wrote:
Hi Bob,

I agree with you that HTTP/2 flow control is not useful for your a) use-case, intermediate buffer control. It is, however, necessary for the b) use case, memory management of an endpoint.

An HTTP server is often divided into a core set of components which manage work scheduling and HTTP protocol aspects, as well as a set of independantly developed  “application” components, that  are often, but not necessarily,  provided by the end-user of the server. There is typically an abstraction between application code and server code, where the application is only ever aware of its stream. The stream may be provided by a single HTTP/1.1 TCP socket, or it might be multiplexed over HTTP/2 or SPDY, or some other protocol.  In all of these cases the abstraction appears the same to the application, as it need not care. The application code typically isn’t required (nor able) to read everything immediately; instead, it is  allowed to read as it processes the data, and this ultimately requires the server to buffer the data in the interim (since other streams should not be blocked). HTTP/2 flow control allows the server to limit the maximum memory usage of this buffering, which would otherwise be significant since HTTP data can be of arbitrary length.

-Jason

On Mar 19, 2015, at 6:54 AM, Bob Briscoe <[hidden email]> wrote:

Patrick,

Thank you for taking the time to read my review carefully. I've been away from mail a few days, which should have allowed time for anything substantive to come from the people with SPDY experience.

We only had the Rob/Greg exchange which merely talked about what some theoretical thing called flow control might be useful for, rather than really addressing my concern that the credit-based protocol provided in h2 is unlikely to be able to provide control if it is needed. Nonetheless, Greg did summarise well the limited usefulness of h2's mechanism.

I should probably have provided a summary of my (long) review, which I try to do below.


I agree that there could be some use for the h2 credit-based protocol at the very start of a stream (in both the C->S and S->C PUSH cases you mentioned). And it might possibly be useful for the cases you mention about unsolicited data, which sound like they occur at the start too. Altho without knowing more about these DoS cases I'm not sure; RST_STREAM might have been sufficient.

However, once a stream has been allowed to open up it's rate, your observation that we mostly see very large windows is what I predicted. It demonstrates that the credit-based protocol does not have sufficient information to be useful to regulate flow. It is effectively being treated like the human appendix - something that no longer serves any purpose but you have to continually put in effort to keep it healthy otherwise it could stop the rest of the system from working.

For this reason, I questioned why flow control has been made mandatory. And I suggested instead that the credit-based flow control in h2 could be
i) mandatory for a data sender to respond to incoming WINDOW_UPDATEs (and therefore a data receiver can gracefully protect itself from DoS by discarding data that exceeds the credit it has previously made available)
ii) optional for a data receiver to emit WINDOW_UPDATEs (i.e. does not even have to implement this part of the code).


Bob

At 21:12 10/03/2015, Patrick McManus wrote:
Hi Bob - I think your comments are appreciated. Its just one of those things where people have dispersed to other things and aren't necessarily in a place to revisit all the ground work again at this stage for a new go round. It was in large part the operational feedback and needs of the google.com team, who has a lot of experience operating spdy at scale, that created the flow control provisions. hopefully those folks will chime in more authoritatively than my musings below:

I'm sure there is quite a bit to learn here - indeed poorly configured use of the window_update mechanism has been (predictably) a source of unintended bottlenecks during both spdy and h2 trials. The spec does try and highlight that there can be dragons here and implementations that don't need the features it can bring should provide essentially infinite credits to steer clear of them.

During the various trials I've seen h2 per stream flow control deployed successfully for a couple of use cases - both of them essentially deal with unsolicited data.

The primary one is essentially a more flexible version of h1's 100-continue. When a client presents a large message body (e.g. a file upload) a multiplexing server needs a way of saying "these are how many buffers I've got available while I figure out where I'm going to sink this incoming data (perhaps to another server I need to connect to)". Presenting this on a per-stream basis allows the server to limit one stream while another (with a distinct sink) can proceed independently.  IMO this value should represent resources available and should be independent of BDP. This is why in practice you see clients with extremely large stream windows - most circumstances just want the data to flow at line rate (as you describe) and aren't trying to invoke flow control. The firefox default window is 256MB per stream - that's not going to slow down the sender nor require frequent window_update generation.

The other use case is when the server pushes resources at a client without them being requested, which is a new feature of h2. This is conceptually similar to the server receiving a POST - suddenly there is a large amount of inbound data that the implementation might not have the resources to store completely. We can't just let TCP flow control take care of the situation because the TCP session is being multiplexed between multiple streams that need to be serviced. In this case the client accepts "some" of the stream based on policy and resource availability and can leave the stream in limbo until an event comes along that tells it to resume the transfer by issuing credits or reject it via cancel.

hth a least a bit.


On Tue, Mar 10, 2015 at 4:31 PM, Bob Briscoe <[hidden email]> wrote:
HTTP/2 folks,
I know extensibility had already been discussed and put to bed, so the WG is entitled to rule out opening healed wounds.
But have points like those I've made about flow control been raised before? Please argue. I may be wrong. Discussion can go on in parallel to the RFC publication process, even tho the process doesn't /require/ you to talk to me.

If I'm right, then implementers are being mandated to write complex flow control code, when it might have little bearing on the performance benefits measured for http/2.
Even if I'm right, and the WG goes ahead anyway, /I/ will understand. My review came in after your deadline.
However, bear in mind that the Webosphere might not be so forgiving. If h2 goes ahead when potential problems have been identified, it could get a bad reputation simply due to the uncertainty, just when you want more people to take it up and try it out. Given you've put in a few person-years of effort, I would have thought you would not want to risk a reputation flop.
I'm trying to help - I just can't go any faster.

Bob

At 14:43 06/03/2015, Bob Briscoe wrote:
HTTP/2 folks,
As I said, consider this as a late review from a clueful but fresh pair of eyes.
My main concerns with the draft are:
* extensibility (previous posting)
* flow control (this posting - apologies for the length - I've tried to explain properly)
* numerous open issues left dangling (see subsequent postings)
===HTTP/2 FLOW CONTROL===
The term 'window' as used throughout is incorrect and highly confusing, in:
* 'flow control window' (44 occurrences),
* 'initial window size' (5),
* or just 'window size' (8)
* and the terms WINDOW_UPDATE and SETTINGS_INITIAL_WINDOW.
The HTTP/2 WINDOW_UPDATE mechanism constrains HTTP/2 to use only credit-based flow control, not window-based. At one point, it actually says it is credit-based (in flow control principle #2 < https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-5.2.1 > ), but otherwise it incorrectly uses the term window.
This is not just an issue of terminology. The more I re-read the flow control sections the more I became convinced that this terminology is not just /confusing/, rather it's evidence of /confusion/. It raises the questions
* "Is HTTP/2 capable of the flow control it says it's capable of?"
* "What type of flow-control protocol ought HTTP/2 to be capable of?"
* "Can the WINDOW_UPDATE frame support the flow-control that HTTP/2 needs?"
To address these questions, it may help if I separate the two different cases HTTP/2 flow control attempts to cover (my own separation, not from the draft):
a) Intermediate buffer control
Here, a stream's flow enters /and/ leaves a buffer (e.g. at the app-layer of an intermediate node).
b) Flow control by the ultimate client app.
Here flow never releases memory (at least not during the life of the connection). The flow is solely consuming more and more memory (e.g. data being rendered into a client app's memory).
==a) Intermediate buffer control==
For this, sliding window-based flow control would be appropriate, because the goal is to keep the e2e pipeline full without wasting buffer.
Let me prove HTTP/2 cannot do window flow control. For window flow control, the sender needs to be able to advance both the leading and trailing edges of the window. In the draft:
* WINDOW_UPDATE frames can only advance the leading edge of a 'window' (and they are constrained to positive values).
* To advance the trailing edge, window flow control would need a continuous stream of acknowledgements back to the sender (like TCP). The draft does not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at the transport layer, so the sending app-layer cannot advance the trailing edge of a 'window'.
So the protocol can only support credit-based flow control. It is incapable of supporting window flow control.
Next, I don't understand how a receiver can set the credit in 'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to answer the question "How much more can I send than I have seen ACK'd?" that would be easy. But because the protocol is restricted to credit, the sender needs the receiver to answer the much harder open-ended question, "How much more can I send?" So the sender needs the receiver to know how many ACKs the sender has seen, but neither of them know that.
The receiver can try, by taking a guess at the bandwidth-delay product, and adjusting the guess up or down, depending on whether its buffer is growing or shrinking. But this only works if the unknown bandwidth-delay product stays constant.
However, BDP will usually be highly variable, as other streams come and go. So, in the time it takes to get a good estimate of the per-stream BDP, it will probably have changed radically, or the stream will most likely have finished anyway. This is why TCP bases flow control on a window, not credit. By complementing window updates with ACK stream info, a TCP sender has sufficient info to control the flow.
The draft is indeed correct when it says:
"   this can lead to suboptimal use of available
   network resources if flow control is enabled without knowledge of the
   bandwidth-delay product (see [RFC7323]).
"
Was this meant to be a veiled criticism of the protocol's own design? A credit-based flow control protocol like that in the draft does not provide sufficient information for either end to estimate the bandwidth-delay product, given it will be varying rapidly.
==b) Control by the ultimate client app==
For this case, I believe neither window nor credit-based flow control is appropriate:
* There is no memory management issue at the client end - even if there's a separate HTTP/2 layer of memory between TCP and the app, it would be pointless to limit the memory used by HTTP/2, because the data is still going to sit in the same user-space memory (or at least about the same amount of memory) when HTTP/2 passes it over for rendering.
* Nonetheless, the receiving client does need to send messages to the sender to supplement stream priorities, by notifying when the state of the receiving application has changed (e.g. if the user's focus switches from one browser tab to another).
* However, credit-based flow control would be very sluggish for such control, because credit cannot be taken back once it has been given (except HTTP/2 allows SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure that hits all streams together).
==Flow control problem summary==
With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.
==Towards a solution?==
I think 'type-a' flow control (for intermediate buffer control) does not need to be at stream-granularity. Indeed, I suspect a proxy could control its app-layer buffering by controlling the receive window of the incoming TCP connection. Has anyone assessed whether this would be sufficient?
I can understand the need for 'type-b' per-stream flow control (by the ultimate client endpoint). Perhaps it would be useful for the receiver to emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream PRIORITY would be sufficient? Either would minimise the response time to a half round trip. Whereas credit flow-control will be much more sluggish (see 'Flow control problem summary').
Either approach would correctly propagate e2e. An intermediate node would naturally tend to prioritise incoming streams that fed into prioritised outgoing streams, so priority updates would tend to propagate from the ultimate receiver, through intermediate nodes, up to the ultimate sender.
==Flow control coverage==
The draft exempts all TCP payload bytes from flow control except HTTP/2 data frames. No rationale is given for this decision. The draft says it's important to manage per-stream memory, then it exempts all the frame types except data, even tho each byte of a non-data frame consumes no less memory than a byte of a data frame.
What message does this put out? "Flow control is not important for one type of bytes with unlimited total size, but flow control is so important that it has to be mandatory for the other type of bytes."
It is certainly critical that WINDOW_UPDATE messages are not covered by flow control, otherwise there would be a real risk of deadlock. It might be that there are dependencies on other frame types that would lead to a dependency loop and deadlock. It would be good to know what the rationale behind these rules was.
==Theory?==
I am concerned that HTTP/2 flow control may have entered new theoretical territory, without suitable proof of safety. The only reassurance we have is one implementation of a flow control algorithm (SPDY), and the anecdotal non-evidence that no-one using SPDY has noticed a deadlock yet (however, is anyone monitoring for deadlocks?).
Whereas SPDY has been an existence proof that an approach like http/2 'works', so far all the flow control algos have been pretty much identical (I think that's true?). I am concerned that the draft takes the InterWeb into uncharted waters, because it allows unconstrained diversity in flow control algos, which is an untested degree of freedom.
The only constraints the draft sets are:
* per-stream flow control is mandatory
* the only protocol message for flow control algos to use is the WINDOW_UPDATE credit message, which cannot be negative
* no constraints on flow control algorithms.
* and all this must work within the outer flow control constraints of TCP.
Some algos might use priority messages to make flow control assumptions. While other algos might associate PRI and WINDOW_UPDATE with different meanings. What confidence do we have that everyone's optimisation algorithms will interoperate? Do we know there will not be certain types of application where deadlock is likely?

"   When using flow
   control, the receiver MUST read from the TCP receive buffer in a
   timely fashion.  Failure to do so could lead to a deadlock when
   critical frames, such as WINDOW_UPDATE, are not read and acted upon.
"
I've been convinced (offlist) that deadlock will not occur as long as the app consumes data 'greedily' from TCP. That has since been articulated in the above normative text. But how sure can we be that every implementer's different interpretations of 'timely' will still prevent deadlock?
Until a good autotuning algorithm for TCP receive window management was developed, good window management code was nearly non-existent. Managing hundreds of interdependent stream buffers is a much harder problem. But implementers are being allowed to just 'Go forth and innovate'. This might work if everyone copies available open source algo(s). But they might not, and they don't have to.
This all seems like 'flying by the seat of the pants'.
==Mandatory Flow Control? ==
"      3. [...] A sender
       MUST respect flow control limits imposed by a receiver."
This ought to be a 'SHOULD' because it is contradicted later - if settings change.
"   6.  Flow control cannot be disabled."
Also effectively contradicted half a page later:
"   Deployments that do not require this capability can advertise a flow
   control window of the maximum size (2^31-1), and by maintaining this
   window by sending a WINDOW_UPDATE frame when any data is received.
   This effectively disables flow control for that receiver."
And contradicted in the definition of half closed (remote):
"  half closed (remote):
      [...] an endpoint is no longer
      obligated to maintain a receiver flow control window.
"
And contradicted in 8.3. The CONNECT Method, which says:
"  Frame types other than DATA
   or stream management frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY)
   MUST NOT be sent on a connected stream, and MUST be treated as a
   stream error (Section 5.4.2) if received.
"
Why is flow control so important that it's mandatory, but so unimportant that you MUST NOT do it when using TLS e2e?
Going back to the earlier quote about using the max window size, it seems perverse for the spec to require endpoints to go through the motions of flow control, even if they arrange for it to affect nothing, but to still require implementation complexity and bandwidth waste with a load of redundant WINDOW_UPDATE frames.
HTTP is used on a wide range of devices, down to the very small and challenged. HTTP/2 might be desirable in such cases, because of the improved efficiency (e.g. header compression), but in many cases the stream model may not be complex enough to need stream flow control.
So why not make flow control optional on the receiving side, but mandatory to implement on the sending side? Then an implementation could have no machinery for tuning window sizes, but it would respond correctly to those set by the other end, which requires much simpler code.
If a receiving implemention chose not to do stream flow control, it could still control flow at the connection (stream 0) level, or at least at the TCP level.

==Inefficiency?==

5.2. Flow Control
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.
==Flow Control Requirements===
I'm not convinced that clear understanding of flow control requirements has driven flow control design decisions.
The draft states various needs for flow-control without giving me a feel of confidence that it has separated out the different cases, and chosen a protocol suitable for each. I tried to go back to the early draft on flow control requirements < http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01 >, and I was not impressed.
I have quoted below the various sentences in the draft that state what flow control is believed to be for. Below that, I have attempted to crystalize out the different concepts, each of which I have tagged within the quotes.
* 2. HTTP/2 Protocol Overview says
  "Flow control and prioritization ensure that it is possible to efficiently use multiplexed streams. [Y]
   Flow control (Section 5.2) helps to ensure that only data that can be used by a receiver is transmitted. [X]"
* 5.2. Flow Control says:
  "Using streams for multiplexing introduces contention over use of the TCP connection [X], resulting in blocked streams [Z]. A flow control scheme ensures that streams on the same connection do not destructively interfere with each other [Z]."
* 5.2.2. Appropriate Use of Flow Control
"  Flow control is defined to protect endpoints that are operating under
   resource constraints.  For example, a proxy needs to share memory
   between many connections, and also might have a slow upstream
   connection and a fast downstream one [Y].  Flow control addresses cases
   where the receiver is unable to process data on one stream, yet wants
   to continue to process other streams in the same connection [X]."
"  Deployments with constrained resources (for example, memory) can
   employ flow control to limit the amount of memory a peer can consume. [Y]
Each requirement has been tagged as follows:
[X] Notification of the receiver's changing utility for each stream
[Y] Prioritisation of streams due to contention over the streaming capacity available to the whole connection.
[Z] Ensuring one stream is not blocked by another.
[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y] sounds more like optimisation across a continuous spectrum.

Regards


Bob
________________________________________________________________
Bob Briscoe,                                                  BT
________________________________________________________________
Bob Briscoe,                                                  BT
________________________________________________________________
Bob Briscoe,                                                  BT

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

________________________________________________________________
Bob Briscoe,                                                  BT


--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Greg Wilkins-3

On 20 March 2015 at 09:28, Roberto Peon <[hidden email]> wrote:
Suffice it to say that the flow control is good enough to prevent oom on servers and intermediaries. It is otherwise imperfect, though simple

The use-case of memory protection is key.    Perhaps the mechanism should have been called Buffer Control rather than Flow Control?

I do not like the suggestion by Bob that it should be optional with excess data discarded, as this would necessitate an acknowledgement protocol and data retention in the sender (thus having a real window).     We have a reliable channel, so we should not throw away it's advantages.  The questions is, how much effective parallelism can we get with a single reliable channel.

I think the confusion here comes back to poor setting of requirements in the charter.   We were tasked with preventing HOL blocking, but there are many ways that can be looked at.   

One view of that requirement is to say that means that that a large stream should never prevent another small stream from proceeding even if the  bandwidth is saturated.    This view requires the flow control mechanism to prevent the TCP flow control from being hit, so that a small frame can always be sent.  Thus we sacrifice maximal throughput for stream fairness.

An alternative view of preventing HOL is that we just need to prevent small streams from waiting for the completion of large streams.  So long as we fragment and interleave our streams, then TCP flow control can be hit and that will delay all streams on the connection, but the stream fragmentation/interleaving will ensure that all frames progress in parallel as fast as the data throughput allows.   This means a sender cannot just suddenly wake up and expect to be able to immediately send a small frame as the connection may be saturated.... but that's always the case even if multiple connections are used!

I think we often write about the mechanism as if we are striving for the former, but I think we have specified the later.  So confusion comes from the poor statement or requirements and the perhaps resulting confused/wrong description of the mechanism.

It appears that rather than a window based flow control mechanism we actually have a credit based buffer control mechanism - which may or may not impact maximal throughput depending on how it is configured and the ratio between available memory and network transfer rates.   We are now engaged in a global experiment to see how that works out (hence I do think the idea of this being an experimental standard is not a bad one).

cheers




--
Greg Wilkins <[hidden email]>  @  Webtide - an Intalio subsidiary
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.
Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Stuart Douglas
In reply to this post by Bob Briscoe


==a) Intermediate buffer control==
For this, sliding window-based flow control would be appropriate, because the goal is to keep the e2e pipeline full without wasting buffer.

Let me prove HTTP/2 cannot do window flow control. For window flow control, the sender needs to be able to advance both the leading and trailing edges of the window. In the draft:
* WINDOW_UPDATE frames can only advance the leading edge of a 'window' (and they are constrained to positive values).
* To advance the trailing edge, window flow control would need a continuous stream of acknowledgements back to the sender (like TCP). The draft does not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at the transport layer, so the sending app-layer cannot advance the trailing edge of a 'window'.

So the protocol can only support credit-based flow control. It is incapable of supporting window flow control.

Next, I don't understand how a receiver can set the credit in 'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to answer the question "How much more can I send than I have seen ACK'd?" that would be easy. But because the protocol is restricted to credit, the sender needs the receiver to answer the much harder open-ended question, "How much more can I send?" So the sender needs the receiver to know how many ACKs the sender has seen, but neither of them know that.

The receiver can try, by taking a guess at the bandwidth-delay product, and adjusting the guess up or down, depending on whether its buffer is growing or shrinking. But this only works if the unknown bandwidth-delay product stays constant.

However, BDP will usually be highly variable, as other streams come and go. So, in the time it takes to get a good estimate of the per-stream BDP, it will probably have changed radically, or the stream will most likely have finished anyway. This is why TCP bases flow control on a window, not credit. By complementing window updates with ACK stream info, a TCP sender has sufficient info to control the flow.

The draft is indeed correct when it says:
"   this can lead to suboptimal use of available
   network resources if flow control is enabled without knowledge of the
   bandwidth-delay product (see [RFC7323]).
"
Was this meant to be a veiled criticism of the protocol's own design? A credit-based flow control protocol like that in the draft does not provide sufficient information for either end to estimate the bandwidth-delay product, given it will be varying rapidly.

From my point of view as a server/proxy implementor the flow control window mostly represent the amount of data I am prepared to buffer. 

From the server as a receiver case we are basically acting as an intermediary between the network and an end user application. In this case the flow control credit basically represents the maximum amount of data that I am prepared to buffer at the HTTP layer. In this case the answer to 'how much more can I send' is simple, it is basically the amount of data I am prepared to buffer. Because I have no idea how quickly (if at all) the user application will consume the data, all I can really do is buffer it and deliver it as the user application requests it. 

If I set the flow control window to larger than I am prepared to buffer and the application does not consume data quickly enough then I have two options, which are basically stop reading (head of line blocking), or reset the stream, neither of which is particularly good (head of line blocking is particularly problematic if the user application is trying to write a response before reading the request, as window updates will not be processed). 

If I am only prepared to buffer a small amount of data then my performance is not going to be great no matter what flow control implementation is in use, and I think that this is basically a limitation of a multiplexed protocol (unless you are prepared to accept potential HOL blocking).

This mostly only affects server uploads, as clients will likely have to buffer the whole response anyway so are not constrained by buffer size. 

All the issues above basically apply to the intermediary/proxy use case as well. 

Basically I guess what I am getting to is that yes, there are some situations where HTTP2 might perform worse than HTTP1, however I think the underlying problem is intrinsic to any sort of multiplexed protocol, rather than HTTP2's flow control mechanism.


==b) Control by the ultimate client app==
For this case, I believe neither window nor credit-based flow control is appropriate:
* There is no memory management issue at the client end - even if there's a separate HTTP/2 layer of memory between TCP and the app, it would be pointless to limit the memory used by HTTP/2, because the data is still going to sit in the same user-space memory (or at least about the same amount of memory) when HTTP/2 passes it over for rendering.

Not necessarily, not all clients are browsers, and even with browsers when downloading a file I imagine the data will generally be transferred straight to disk rather staying in memory. In general I agree with you though, and I think most clients will want to set a large window size.
 
* Nonetheless, the receiving client does need to send messages to the sender to supplement stream priorities, by notifying when the state of the receiving application has changed (e.g. if the user's focus switches from one browser tab to another).
* However, credit-based flow control would be very sluggish for such control, because credit cannot be taken back once it has been given (except HTTP/2 allows SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure that hits all streams together).

This support is provided by PRIORITY frames, using flow control for such use cases is very problematic and has a good chance of leading to deadlocks. For example consider a Java Servlet container, new requests will be read from the underlying connection, and then dispatched to a worker thread to generate the page. Once a server has started processing a request there is no way to pause the request in a way that frees up the resources (threads, database connections etc) in use. I think almost every use case more complex than simple file serving has this issue, once a server has started processing a request there is generally no way to suspend it and free the resources. 

So in your example if the user switches tabs and you stop sending window updates for streams in use by the old browser tab, while sending new requests for the new tab it is possible the server will be in a situation where it is not prepared to allocate resources for the new requests until the existing ones are complete, however these will never complete as the browser has stopped sending window updates. 

You could argue that a server should limit the max streams value to the number of streams that it is prepared to allocate resources for, however this greatly limits the utility of the priority mechanism, as it means that servers will always handle requests on a first come first served basis. If we set the maximum streams value to a higher amount than what we are prepared to allocate resources for and queue requests then when a request finishes we can pick the highest priority request from the queue to allocate resources to (not to mention there is no round trip delay because the request is queued).

Basically IMHO flow control should not be used to control priority, that is what PRIORITY frames are for, and in general servers can only ever do priority on a best effort approach anyway. If you try and use flow control to enforce a strict priority mechanism you run a very real risk of deadlocks.
 

==Flow control problem summary==

With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.

For the reasons I outlined above I don't think this is actually a problem. Also in terms of clawing back credit I thought that in general it was a bad idea? The TCP RFC explicitly states that "shrinking the window" is strongly discouraged, although I must admit I am not fully aware of the reasoning.

Stuart
 

==Towards a solution?==

I think 'type-a' flow control (for intermediate buffer control) does not need to be at stream-granularity. Indeed, I suspect a proxy could control its app-layer buffering by controlling the receive window of the incoming TCP connection. Has anyone assessed whether this would be sufficient?

I can understand the need for 'type-b' per-stream flow control (by the ultimate client endpoint). Perhaps it would be useful for the receiver to emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream PRIORITY would be sufficient? Either would minimise the response time to a half round trip. Whereas credit flow-control will be much more sluggish (see 'Flow control problem summary').

Either approach would correctly propagate e2e. An intermediate node would naturally tend to prioritise incoming streams that fed into prioritised outgoing streams, so priority updates would tend to propagate from the ultimate receiver, through intermediate nodes, up to the ultimate sender.

==Flow control coverage==

The draft exempts all TCP payload bytes from flow control except HTTP/2 data frames. No rationale is given for this decision. The draft says it's important to manage per-stream memory, then it exempts all the frame types except data, even tho each byte of a non-data frame consumes no less memory than a byte of a data frame.

What message does this put out? "Flow control is not important for one type of bytes with unlimited total size, but flow control is so important that it has to be mandatory for the other type of bytes."

It is certainly critical that WINDOW_UPDATE messages are not covered by flow control, otherwise there would be a real risk of deadlock. It might be that there are dependencies on other frame types that would lead to a dependency loop and deadlock. It would be good to know what the rationale behind these rules was.

I think a lot of people had similar concerns. There is a discussion about it here: https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0647.html
 

==Theory?==

I am concerned that HTTP/2 flow control may have entered new theoretical territory, without suitable proof of safety. The only reassurance we have is one implementation of a flow control algorithm (SPDY), and the anecdotal non-evidence that no-one using SPDY has noticed a deadlock yet (however, is anyone monitoring for deadlocks?).

Whereas SPDY has been an existence proof that an approach like http/2 'works', so far all the flow control algos have been pretty much identical (I think that's true?). I am concerned that the draft takes the InterWeb into uncharted waters, because it allows unconstrained diversity in flow control algos, which is an untested degree of freedom.

The only constraints the draft sets are:
* per-stream flow control is mandatory
* the only protocol message for flow control algos to use is the WINDOW_UPDATE credit message, which cannot be negative
* no constraints on flow control algorithms.
* and all this must work within the outer flow control constraints of TCP.

Some algos might use priority messages to make flow control assumptions. While other algos might associate PRI and WINDOW_UPDATE with different meanings. What confidence do we have that everyone's optimisation algorithms will interoperate? Do we know there will not be certain types of application where deadlock is likely?

"   When using flow
   control, the receiver MUST read from the TCP receive buffer in a
   timely fashion.  Failure to do so could lead to a deadlock when
   critical frames, such as WINDOW_UPDATE, are not read and acted upon.
"
I've been convinced (offlist) that deadlock will not occur as long as the app consumes data 'greedily' from TCP. That has since been articulated in the above normative text. But how sure can we be that every implementer's different interpretations of 'timely' will still prevent deadlock?

Until a good autotuning algorithm for TCP receive window management was developed, good window management code was nearly non-existent. Managing hundreds of interdependent stream buffers is a much harder problem. But implementers are being allowed to just 'Go forth and innovate'. This might work if everyone copies available open source algo(s). But they might not, and they don't have to.

This all seems like 'flying by the seat of the pants'.

==Mandatory Flow Control? ==

"      3. [...] A sender
       MUST respect flow control limits imposed by a receiver."
This ought to be a 'SHOULD' because it is contradicted later - if settings change.

"   6.  Flow control cannot be disabled."
Also effectively contradicted half a page later:
"   Deployments that do not require this capability can advertise a flow
   control window of the maximum size (2^31-1), and by maintaining this
   window by sending a WINDOW_UPDATE frame when any data is received.
   This effectively disables flow control for that receiver."

And contradicted in the definition of half closed (remote):
"  half closed (remote):
      [...] an endpoint is no longer
      obligated to maintain a receiver flow control window.
"

And contradicted in 8.3. The CONNECT Method, which says:
"  Frame types other than DATA
   or stream management frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY)
   MUST NOT be sent on a connected stream, and MUST be treated as a
   stream error (Section 5.4.2) if received.
"

Why is flow control so important that it's mandatory, but so unimportant that you MUST NOT do it when using TLS e2e?

Going back to the earlier quote about using the max window size, it seems perverse for the spec to require endpoints to go through the motions of flow control, even if they arrange for it to affect nothing, but to still require implementation complexity and bandwidth waste with a load of redundant WINDOW_UPDATE frames.

HTTP is used on a wide range of devices, down to the very small and challenged. HTTP/2 might be desirable in such cases, because of the improved efficiency (e.g. header compression), but in many cases the stream model may not be complex enough to need stream flow control.

So why not make flow control optional on the receiving side, but mandatory to implement on the sending side? Then an implementation could have no machinery for tuning window sizes, but it would respond correctly to those set by the other end, which requires much simpler code.

If a receiving implemention chose not to do stream flow control, it could still control flow at the connection (stream 0) level, or at least at the TCP level.


==Inefficiency?==

5.2. Flow Control
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.

==Flow Control Requirements===

I'm not convinced that clear understanding of flow control requirements has driven flow control design decisions.

The draft states various needs for flow-control without giving me a feel of confidence that it has separated out the different cases, and chosen a protocol suitable for each. I tried to go back to the early draft on flow control requirements < http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01 >, and I was not impressed.

I have quoted below the various sentences in the draft that state what flow control is believed to be for. Below that, I have attempted to crystalize out the different concepts, each of which I have tagged within the quotes.

* 2. HTTP/2 Protocol Overview says
  "Flow control and prioritization ensure that it is possible to efficiently use multiplexed streams. [Y]
   Flow control (Section 5.2) helps to ensure that only data that can be used by a receiver is transmitted. [X]"
* 5.2. Flow Control says:
  "Using streams for multiplexing introduces contention over use of the TCP connection [X], resulting in blocked streams [Z]. A flow control scheme ensures that streams on the same connection do not destructively interfere with each other [Z]."
* 5.2.2. Appropriate Use of Flow Control
"  Flow control is defined to protect endpoints that are operating under
   resource constraints.  For example, a proxy needs to share memory
   between many connections, and also might have a slow upstream
   connection and a fast downstream one [Y].  Flow control addresses cases
   where the receiver is unable to process data on one stream, yet wants
   to continue to process other streams in the same connection [X]."
"  Deployments with constrained resources (for example, memory) can
   employ flow control to limit the amount of memory a peer can consume. [Y]

Each requirement has been tagged as follows:
[X] Notification of the receiver's changing utility for each stream
[Y] Prioritisation of streams due to contention over the streaming capacity available to the whole connection.
[Z] Ensuring one stream is not blocked by another.

[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y] sounds more like optimisation across a continuous spectrum.


Regards



Bob

________________________________________________________________
Bob Briscoe,                                                  BT

Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

David Krauss

On 2015–03–20, at 8:18 AM, Stuart Douglas <[hidden email]> wrote:

Basically IMHO flow control should not be used to control priority, that is what PRIORITY frames are for, and in general servers can only ever do priority on a best effort approach anyway. If you try and use flow control to enforce a strict priority mechanism you run a very real risk of deadlocks.

When I made similar arguments last year, the only reply was that flow control throttling and negative credit work and nobody wants to fix what isn’t broken. Also, nobody wants to add complexity to PRIORITY that would overlap functionality already provided by WINDOW_UPDATE. Theoretical arguments about deadlocks aren’t enough, you have to actually implement and demonstrate the condition.

This standardization process is based on rough consensus and working code, not theorizing. However, the definition of “working” is left up to the prototypers who are forming the consensus, which can lead to circular logic. At any rate, the ship has sailed. What we can do now is do our best to implement robustness over performance, and publicly demonstrate any apparent weaknesses and exploits.


On 2015–03–06, at 10:43 PM, Bob Briscoe <[hidden email]> wrote:

5.2. Flow Control 
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.

The decision in this case was that efficiency is not a concern.

I’ve not reviewed this thread fully, sorry, but you might find insight in the mailing list archives. These controversies came up during the evolutionary process.
Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Bob Briscoe
In reply to this post by Jason Greene
Jason,

At 16:00 19/03/2015, Jason Greene wrote:
I think thats a good argument for why it’s not suitable for rate-limiting relative to a variable bandwidth product of a single connection (which i took as your use-case a).

I believe there is no need for an intermediate node to do flow control for individual streams. It does need to control the whole envelope within which all the streams are flowing through the proxy's app-layer buffer memory (e.g. due to a thick incoming pipe feeding a thin outgoing pipe). The best mechanism for controlling the app-layer buffer consumption of the aggregate connection is for the intermediate node to control the TCP receive window of the incoming stream.

That doesn't preclude the intermediate node passing on any per-stream flow control messages emanating from the ultimate receiver so that the ultimate sender controls each stream's rate, which will alter the balance between streams within the overall envelope at the proxy.

I explained all this in my review.

 However, memory limits are typically constant, and an HTTP/2 server can be configured to limit the capacity of the server based on the combination of the “window” size, the number of streams allowed, and the number of connections allowed. Without a hard data credit, the server would have to either HOL block, prevent valid large requests from processing, or run out of resources and start dropping requests.

I think you've moved away from the intermediate node case here? Your suggestions are valid. They are the crude controls available because more fine-grained per-stream control seems unattainable in h2, even tho I think the impression has been given that this requirement had been satisfied.

Altho per-stream flow control seems unattainable, there is still stream priority.



Bob


On Mar 19, 2015, at 10:38 AM, Bob Briscoe <[hidden email]> wrote:

Jason,

Yes, I agree this is what we /want/ per-stream flow control to do. My review explained why the flow control in h2 won't be able to do this.

Specifically:

==Flow control problem summary==
With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.
IOW: Between a rock and a hard place,... but don't tell them where the rock is.


Bob

At 15:08 19/03/2015, Jason Greene wrote:
Hi Bob,

I agree with you that HTTP/2 flow control is not useful for your a) use-case, intermediate buffer control. It is, however, necessary for the b) use case, memory management of an endpoint.

An HTTP server is often divided into a core set of components which manage work scheduling and HTTP protocol aspects, as well as a set of independantly developed  “application” components, that  are often, but not necessarily,  provided by the end-user of the server. There is typically an abstraction between application code and server code, where the application is only ever aware of its stream. The stream may be provided by a single HTTP/1.1 TCP socket, or it might be multiplexed over HTTP/2 or SPDY, or some other protocol.  In all of these cases the abstraction appears the same to the application, as it need not care. The application code typically isn’t required (nor able) to read everything immediately; instead, it is  allowed to read as it processes the data, and this ultimately requires the server to buffer the data in the interim (since other streams should not be blocked). HTTP/2 flow control allows the server to limit the maximum memory usage of this buffering, which would otherwise be significant since HTTP data can be of arbitrary length.

-Jason

On Mar 19, 2015, at 6:54 AM, Bob Briscoe <[hidden email]> wrote:

Patrick,

Thank you for taking the time to read my review carefully. I've been away from mail a few days, which should have allowed time for anything substantive to come from the people with SPDY experience.

We only had the Rob/Greg exchange which merely talked about what some theoretical thing called flow control might be useful for, rather than really addressing my concern that the credit-based protocol provided in h2 is unlikely to be able to provide control if it is needed. Nonetheless, Greg did summarise well the limited usefulness of h2's mechanism.

I should probably have provided a summary of my (long) review, which I try to do below.


I agree that there could be some use for the h2 credit-based protocol at the very start of a stream (in both the C->S and S->C PUSH cases you mentioned). And it might possibly be useful for the cases you mention about unsolicited data, which sound like they occur at the start too. Altho without knowing more about these DoS cases I'm not sure; RST_STREAM might have been sufficient.

However, once a stream has been allowed to open up it's rate, your observation that we mostly see very large windows is what I predicted. It demonstrates that the credit-based protocol does not have sufficient information to be useful to regulate flow. It is effectively being treated like the human appendix - something that no longer serves any purpose but you have to continually put in effort to keep it healthy otherwise it could stop the rest of the system from working.

For this reason, I questioned why flow control has been made mandatory. And I suggested instead that the credit-based flow control in h2 could be
i) mandatory for a data sender to respond to incoming WINDOW_UPDATEs (and therefore a data receiver can gracefully protect itself from DoS by discarding data that exceeds the credit it has previously made available)
ii) optional for a data receiver to emit WINDOW_UPDATEs (i.e. does not even have to implement this part of the code).


Bob

At 21:12 10/03/2015, Patrick McManus wrote:
Hi Bob - I think your comments are appreciated. Its just one of those things where people have dispersed to other things and aren't necessarily in a place to revisit all the ground work again at this stage for a new go round. It was in large part the operational feedback and needs of the google.com team, who has a lot of experience operating spdy at scale, that created the flow control provisions. hopefully those folks will chime in more authoritatively than my musings below:

I'm sure there is quite a bit to learn here - indeed poorly configured use of the window_update mechanism has been (predictably) a source of unintended bottlenecks during both spdy and h2 trials. The spec does try and highlight that there can be dragons here and implementations that don't need the features it can bring should provide essentially infinite credits to steer clear of them.

During the various trials I've seen h2 per stream flow control deployed successfully for a couple of use cases - both of them essentially deal with unsolicited data.

The primary one is essentially a more flexible version of h1's 100-continue. When a client presents a large message body (e.g. a file upload) a multiplexing server needs a way of saying "these are how many buffers I've got available while I figure out where I'm going to sink this incoming data (perhaps to another server I need to connect to)". Presenting this on a per-stream basis allows the server to limit one stream while another (with a distinct sink) can proceed independently.  IMO this value should represent resources available and should be independent of BDP. This is why in practice you see clients with extremely large stream windows - most circumstances just want the data to flow at line rate (as you describe) and aren't trying to invoke flow control. The firefox default window is 256MB per stream - that's not going to slow down the sender nor require frequent window_update generation.

The other use case is when the server pushes resources at a client without them being requested, which is a new feature of h2. This is conceptually similar to the server receiving a POST - suddenly there is a large amount of inbound data that the implementation might not have the resources to store completely. We can't just let TCP flow control take care of the situation because the TCP session is being multiplexed between multiple streams that need to be serviced. In this case the client accepts "some" of the stream based on policy and resource availability and can leave the stream in limbo until an event comes along that tells it to resume the transfer by issuing credits or reject it via cancel.

hth a least a bit.


On Tue, Mar 10, 2015 at 4:31 PM, Bob Briscoe <[hidden email]> wrote:
HTTP/2 folks,
I know extensibility had already been discussed and put to bed, so the WG is entitled to rule out opening healed wounds.
But have points like those I've made about flow control been raised before? Please argue. I may be wrong. Discussion can go on in parallel to the RFC publication process, even tho the process doesn't /require/ you to talk to me.
If I'm right, then implementers are being mandated to write complex flow control code, when it might have little bearing on the performance benefits measured for http/2.
Even if I'm right, and the WG goes ahead anyway, /I/ will understand. My review came in after your deadline.
However, bear in mind that the Webosphere might not be so forgiving. If h2 goes ahead when potential problems have been identified, it could get a bad reputation simply due to the uncertainty, just when you want more people to take it up and try it out. Given you've put in a few person-years of effort, I would have thought you would not want to risk a reputation flop.
I'm trying to help - I just can't go any faster.
Bob
At 14:43 06/03/2015, Bob Briscoe wrote:
HTTP/2 folks,
As I said, consider this as a late review from a clueful but fresh pair of eyes.
My main concerns with the draft are:
* extensibility (previous posting)
* flow control (this posting - apologies for the length - I've tried to explain properly)
* numerous open issues left dangling (see subsequent postings)
===HTTP/2 FLOW CONTROL===
The term 'window' as used throughout is incorrect and highly confusing, in:
* 'flow control window' (44 occurrences),
* 'initial window size' (5),
* or just 'window size' (8)
* and the terms WINDOW_UPDATE and SETTINGS_INITIAL_WINDOW.
The HTTP/2 WINDOW_UPDATE mechanism constrains HTTP/2 to use only credit-based flow control, not window-based. At one point, it actually says it is credit-based (in flow control principle #2 < https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-5.2.1 > ), but otherwise it incorrectly uses the term window.
This is not just an issue of terminology. The more I re-read the flow control sections the more I became convinced that this terminology is not just /confusing/, rather it's evidence of /confusion/. It raises the questions
* "Is HTTP/2 capable of the flow control it says it's capable of?"
* "What type of flow-control protocol ought HTTP/2 to be capable of?"
* "Can the WINDOW_UPDATE frame support the flow-control that HTTP/2 needs?"
To address these questions, it may help if I separate the two different cases HTTP/2 flow control attempts to cover (my own separation, not from the draft):
a) Intermediate buffer control
Here, a stream's flow enters /and/ leaves a buffer (e.g. at the app-layer of an intermediate node).
b) Flow control by the ultimate client app.
Here flow never releases memory (at least not during the life of the connection). The flow is solely consuming more and more memory (e.g. data being rendered into a client app's memory).
==a) Intermediate buffer control==
For this, sliding window-based flow control would be appropriate, because the goal is to keep the e2e pipeline full without wasting buffer.
Let me prove HTTP/2 cannot do window flow control. For window flow control, the sender needs to be able to advance both the leading and trailing edges of the window. In the draft:
* WINDOW_UPDATE frames can only advance the leading edge of a 'window' (and they are constrained to positive values).
* To advance the trailing edge, window flow control would need a continuous stream of acknowledgements back to the sender (like TCP). The draft does not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at the transport layer, so the sending app-layer cannot advance the trailing edge of a 'window'.
So the protocol can only support credit-based flow control. It is incapable of supporting window flow control.
Next, I don't understand how a receiver can set the credit in 'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to answer the question "How much more can I send than I have seen ACK'd?" that would be easy. But because the protocol is restricted to credit, the sender needs the receiver to answer the much harder open-ended question, "How much more can I send?" So the sender needs the receiver to know how many ACKs the sender has seen, but neither of them know that.
The receiver can try, by taking a guess at the bandwidth-delay product, and adjusting the guess up or down, depending on whether its buffer is growing or shrinking. But this only works if the unknown bandwidth-delay product stays constant.
However, BDP will usually be highly variable, as other streams come and go. So, in the time it takes to get a good estimate of the per-stream BDP, it will probably have changed radically, or the stream will most likely have finished anyway. This is why TCP bases flow control on a window, not credit. By complementing window updates with ACK stream info, a TCP sender has sufficient info to control the flow.
The draft is indeed correct when it says:
"   this can lead to suboptimal use of available
   network resources if flow control is enabled without knowledge of the
   bandwidth-delay product (see [RFC7323]).
"
Was this meant to be a veiled criticism of the protocol's own design? A credit-based flow control protocol like that in the draft does not provide sufficient information for either end to estimate the bandwidth-delay product, given it will be varying rapidly.
==b) Control by the ultimate client app==
For this case, I believe neither window nor credit-based flow control is appropriate:
* There is no memory management issue at the client end - even if there's a separate HTTP/2 layer of memory between TCP and the app, it would be pointless to limit the memory used by HTTP/2, because the data is still going to sit in the same user-space memory (or at least about the same amount of memory) when HTTP/2 passes it over for rendering.
* Nonetheless, the receiving client does need to send messages to the sender to supplement stream priorities, by notifying when the state of the receiving application has changed (e.g. if the user's focus switches from one browser tab to another).
* However, credit-based flow control would be very sluggish for such control, because credit cannot be taken back once it has been given (except HTTP/2 allows SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure that hits all streams together).
==Flow control problem summary==
With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.
IOW: Between a rock and a hard place,... but don't tell them where the rock is.
==Towards a solution?==
I think 'type-a' flow control (for intermediate buffer control) does not need to be at stream-granularity. Indeed, I suspect a proxy could control its app-layer buffering by controlling the receive window of the incoming TCP connection. Has anyone assessed whether this would be sufficient?
I can understand the need for 'type-b' per-stream flow control (by the ultimate client endpoint). Perhaps it would be useful for the receiver to emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream PRIORITY would be sufficient? Either would minimise the response time to a half round trip. Whereas credit flow-control will be much more sluggish (see 'Flow control problem summary').
Either approach would correctly propagate e2e. An intermediate node would naturally tend to prioritise incoming streams that fed into prioritised outgoing streams, so priority updates would tend to propagate from the ultimate receiver, through intermediate nodes, up to the ultimate sender.
==Flow control coverage==
The draft exempts all TCP payload bytes from flow control except HTTP/2 data frames. No rationale is given for this decision. The draft says it's important to manage per-stream memory, then it exempts all the frame types except data, even tho each byte of a non-data frame consumes no less memory than a byte of a data frame.
What message does this put out? "Flow control is not important for one type of bytes with unlimited total size, but flow control is so important that it has to be mandatory for the other type of bytes."
It is certainly critical that WINDOW_UPDATE messages are not covered by flow control, otherwise there would be a real risk of deadlock. It might be that there are dependencies on other frame types that would lead to a dependency loop and deadlock. It would be good to know what the rationale behind these rules was.
==Theory?==
I am concerned that HTTP/2 flow control may have entered new theoretical territory, without suitable proof of safety. The only reassurance we have is one implementation of a flow control algorithm (SPDY), and the anecdotal non-evidence that no-one using SPDY has noticed a deadlock yet (however, is anyone monitoring for deadlocks?).
Whereas SPDY has been an existence proof that an approach like http/2 'works', so far all the flow control algos have been pretty much identical (I think that's true?). I am concerned that the draft takes the InterWeb into uncharted waters, because it allows unconstrained diversity in flow control algos, which is an untested degree of freedom.
The only constraints the draft sets are:
* per-stream flow control is mandatory
* the only protocol message for flow control algos to use is the WINDOW_UPDATE credit message, which cannot be negative
* no constraints on flow control algorithms.
* and all this must work within the outer flow control constraints of TCP.
Some algos might use priority messages to make flow control assumptions. While other algos might associate PRI and WINDOW_UPDATE with different meanings. What confidence do we have that everyone's optimisation algorithms will interoperate? Do we know there will not be certain types of application where deadlock is likely?
"   When using flow
   control, the receiver MUST read from the TCP receive buffer in a
   timely fashion.  Failure to do so could lead to a deadlock when
   critical frames, such as WINDOW_UPDATE, are not read and acted upon.
"
I've been convinced (offlist) that deadlock will not occur as long as the app consumes data 'greedily' from TCP. That has since been articulated in the above normative text. But how sure can we be that every implementer's different interpretations of 'timely' will still prevent deadlock?
Until a good autotuning algorithm for TCP receive window management was developed, good window management code was nearly non-existent. Managing hundreds of interdependent stream buffers is a much harder problem. But implementers are being allowed to just 'Go forth and innovate'. This might work if everyone copies available open source algo(s). But they might not, and they don't have to.
This all seems like 'flying by the seat of the pants'.
==Mandatory Flow Control? ==
"      3. [...] A sender
       MUST respect flow control limits imposed by a receiver."
This ought to be a 'SHOULD' because it is contradicted later - if settings change.
"   6.  Flow control cannot be disabled."
Also effectively contradicted half a page later:
"   Deployments that do not require this capability can advertise a flow
   control window of the maximum size (2^31-1), and by maintaining this
   window by sending a WINDOW_UPDATE frame when any data is received.
   This effectively disables flow control for that receiver."
And contradicted in the definition of half closed (remote):
"  half closed (remote):
      [...] an endpoint is no longer
      obligated to maintain a receiver flow control window.
"
And contradicted in 8.3. The CONNECT Method, which says:
"  Frame types other than DATA
   or stream management frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY)
   MUST NOT be sent on a connected stream, and MUST be treated as a
   stream error (Section 5.4.2) if received.
"
Why is flow control so important that it's mandatory, but so unimportant that you MUST NOT do it when using TLS e2e?
Going back to the earlier quote about using the max window size, it seems perverse for the spec to require endpoints to go through the motions of flow control, even if they arrange for it to affect nothing, but to still require implementation complexity and bandwidth waste with a load of redundant WINDOW_UPDATE frames.
HTTP is used on a wide range of devices, down to the very small and challenged. HTTP/2 might be desirable in such cases, because of the improved efficiency (e.g. header compression), but in many cases the stream model may not be complex enough to need stream flow control.
So why not make flow control optional on the receiving side, but mandatory to implement on the sending side? Then an implementation could have no machinery for tuning window sizes, but it would respond correctly to those set by the other end, which requires much simpler code.
If a receiving implemention chose not to do stream flow control, it could still control flow at the connection (stream 0) level, or at least at the TCP level.
==Inefficiency?==

5.2. Flow Control
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.
==Flow Control Requirements===
I'm not convinced that clear understanding of flow control requirements has driven flow control design decisions.
The draft states various needs for flow-control without giving me a feel of confidence that it has separated out the different cases, and chosen a protocol suitable for each. I tried to go back to the early draft on flow control requirements < http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01 >, and I was not impressed.
I have quoted below the various sentences in the draft that state what flow control is believed to be for. Below that, I have attempted to crystalize out the different concepts, each of which I have tagged within the quotes.
* 2. HTTP/2 Protocol Overview says
  "Flow control and prioritization ensure that it is possible to efficiently use multiplexed streams. [Y]
   Flow control (Section 5.2) helps to ensure that only data that can be used by a receiver is transmitted. [X]"
* 5.2. Flow Control says:
  "Using streams for multiplexing introduces contention over use of the TCP connection [X], resulting in blocked streams [Z]. A flow control scheme ensures that streams on the same connection do not destructively interfere with each other [Z]."
* 5.2.2. Appropriate Use of Flow Control
"  Flow control is defined to protect endpoints that are operating under
   resource constraints.  For example, a proxy needs to share memory
   between many connections, and also might have a slow upstream
   connection and a fast downstream one [Y].  Flow control addresses cases
   where the receiver is unable to process data on one stream, yet wants
   to continue to process other streams in the same connection [X]."
"  Deployments with constrained resources (for example, memory) can
   employ flow control to limit the amount of memory a peer can consume. [Y]
Each requirement has been tagged as follows:
[X] Notification of the receiver's changing utility for each stream
[Y] Prioritisation of streams due to contention over the streaming capacity available to the whole connection.
[Z] Ensuring one stream is not blocked by another.
[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y] sounds more like optimisation across a continuous spectrum.
Regards

Bob
________________________________________________________________
Bob Briscoe,                                                  BT
________________________________________________________________
Bob Briscoe,                                                  BT
________________________________________________________________
Bob Briscoe,                                                  BT

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

________________________________________________________________
Bob Briscoe,                                                  BT

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

________________________________________________________________
Bob Briscoe,                                                  BT

Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Roberto Peon-2
The appetite for complexity was very low here, and so what we have is the least complex not-obviously-broken functionality that seems likely to prevent HOL blocking.
-=R

On Fri, Mar 20, 2015 at 9:50 AM, Bob Briscoe <[hidden email]> wrote:
Jason,

At 16:00 19/03/2015, Jason Greene wrote:
I think thats a good argument for why it’s not suitable for rate-limiting relative to a variable bandwidth product of a single connection (which i took as your use-case a).

I believe there is no need for an intermediate node to do flow control for individual streams. It does need to control the whole envelope within which all the streams are flowing through the proxy's app-layer buffer memory (e.g. due to a thick incoming pipe feeding a thin outgoing pipe). The best mechanism for controlling the app-layer buffer consumption of the aggregate connection is for the intermediate node to control the TCP receive window of the incoming stream.

That doesn't preclude the intermediate node passing on any per-stream flow control messages emanating from the ultimate receiver so that the ultimate sender controls each stream's rate, which will alter the balance between streams within the overall envelope at the proxy.

I explained all this in my review.

 However, memory limits are typically constant, and an HTTP/2 server can be configured to limit the capacity of the server based on the combination of the “window†size, the number of streams allowed, and the number of connections allowed. Without a hard data credit, the server would have to either HOL block, prevent valid large requests from processing, or run out of resources and start dropping requests.

I think you've moved away from the intermediate node case here? Your suggestions are valid. They are the crude controls available because more fine-grained per-stream control seems unattainable in h2, even tho I think the impression has been given that this requirement had been satisfied.

Altho per-stream flow control seems unattainable, there is still stream priority.



Bob


On Mar 19, 2015, at 10:38 AM, Bob Briscoe <[hidden email]> wrote:

Jason,

Yes, I agree this is what we /want/ per-stream flow control to do. My review explained why the flow control in h2 won't be able to do this.

Specifically:

==Flow control problem summary==
With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.
IOW: Between a rock and a hard place,... but don't tell them where the rock is.


Bob

At 15:08 19/03/2015, Jason Greene wrote:
Hi Bob,

I agree with you that HTTP/2 flow control is not useful for your a) use-case, intermediate buffer control. It is, however, necessary for the b) use case, memory management of an endpoint.

An HTTP server is often divided into a core set of components which manage work scheduling and HTTP protocol aspects, as well as a set of independantly developed  “application” components, that  are often, but not necessarily,  provided by the end-user of the server. There is typically an abstraction between application code and server code, where the application is only ever aware of its stream. The stream may be provided by a single HTTP/1.1 TCP socket, or it might be multiplexed over HTTP/2 or SPDY, or some other protocol.  In all of these cases the abstraction appears the same to the application, as it need not care. The application code typically isnÂ’t required (nor able) to read everything immediately; instead, it is  allowed to read as it processes the data, and this ultimately requires the server to buffer the data in the interim (since other streams should not be blocked). HTTP/2 flow control allows the server to limit the maximum memory usage of this buffering, which would otherwise be significant since HTTP data can be of arbitrary length.

-Jason

On Mar 19, 2015, at 6:54 AM, Bob Briscoe <[hidden email]> wrote:

Patrick,

Thank you for taking the time to read my review carefully. I've been away from mail a few days, which should have allowed time for anything substantive to come from the people with SPDY experience.

We only had the Rob/Greg exchange which merely talked about what some theoretical thing called flow control might be useful for, rather than really addressing my concern that the credit-based protocol provided in h2 is unlikely to be able to provide control if it is needed. Nonetheless, Greg did summarise well the limited usefulness of h2's mechanism.

I should probably have provided a summary of my (long) review, which I try to do below.


I agree that there could be some use for the h2 credit-based protocol at the very start of a stream (in both the C->S and S->C PUSH cases you mentioned). And it might possibly be useful for the cases you mention about unsolicited data, which sound like they occur at the start too. Altho without knowing more about these DoS cases I'm not sure; RST_STREAM might have been sufficient.

However, once a stream has been allowed to open up it's rate, your observation that we mostly see very large windows is what I predicted. It demonstrates that the credit-based protocol does not have sufficient information to be useful to regulate flow. It is effectively being treated like the human appendix - something that no longer serves any purpose but you have to continually put in effort to keep it healthy otherwise it could stop the rest of the system from working.

For this reason, I questioned why flow control has been made mandatory. And I suggested instead that the credit-based flow control in h2 could be
i) mandatory for a data sender to respond to incoming WINDOW_UPDATEs (and therefore a data receiver can gracefully protect itself from DoS by discarding data that exceeds the credit it has previously made available)
ii) optional for a data receiver to emit WINDOW_UPDATEs (i.e. does not even have to implement this part of the code).


Bob

At 21:12 10/03/2015, Patrick McManus wrote:
Hi Bob - I think your comments are appreciated. Its just one of those things where people have dispersed to other things and aren't necessarily in a place to revisit all the ground work again at this stage for a new go round. It was in large part the operational feedback and needs of the google.com team, who has a lot of experience operating spdy at scale, that created the flow control provisions. hopefully those folks will chime in more authoritatively than my musings below:

I'm sure there is quite a bit to learn here - indeed poorly configured use of the window_update mechanism has been (predictably) a source of unintended bottlenecks during both spdy and h2 trials. The spec does try and highlight that there can be dragons here and implementations that don't need the features it can bring should provide essentially infinite credits to steer clear of them.

During the various trials I've seen h2 per stream flow control deployed successfully for a couple of use cases - both of them essentially deal with unsolicited data.

The primary one is essentially a more flexible version of h1's 100-continue. When a client presents a large message body (e.g. a file upload) a multiplexing server needs a way of saying "these are how many buffers I've got available while I figure out where I'm going to sink this incoming data (perhaps to another server I need to connect to)". Presenting this on a per-stream basis allows the server to limit one stream while another (with a distinct sink) can proceed independently.  IMO this value should represent resources available and should be independent of BDP. This is why in practice you see clients with extremely large stream windows - most circumstances just want the data to flow at line rate (as you describe) and aren't trying to invoke flow control. The firefox default window is 256MB per stream - that's not going to slow down the sender nor require frequent window_update generation.


The other use case is when the server pushes resources at a client without them being requested, which is a new feature of h2. This is conceptually similar to the server receiving a POST - suddenly there is a large amount of inbound data that the implementation might not have the resources to store completely. We can't just let TCP flow control take care of the situation because the TCP session is being multiplexed between multiple streams that need to be serviced. In this case the client accepts "some" of the stream based on policy and resource availability and can leave the stream in limbo until an event comes along that tells it to resume the transfer by issuing credits or reject it via cancel.

hth a least a bit.


On Tue, Mar 10, 2015 at 4:31 PM, Bob Briscoe <[hidden email]> wrote:
HTTP/2 folks,
I know extensibility had already been discussed and put to bed, so the WG is entitled to rule out opening healed wounds.
But have points like those I've made about flow control been raised before? Please argue. I may be wrong. Discussion can go on in parallel to the RFC publication process, even tho the process doesn't /require/ you to talk to me.
If I'm right, then implementers are being mandated to write complex flow control code, when it might have little bearing on the performance benefits measured for http/2.
Even if I'm right, and the WG goes ahead anyway, /I/ will understand. My review came in after your deadline.
However, bear in mind that the Webosphere might not be so forgiving. If h2 goes ahead when potential problems have been identified, it could get a bad reputation simply due to the uncertainty, just when you want more people to take it up and try it out. Given you've put in a few person-years of effort, I would have thought you would not want to risk a reputation flop.
I'm trying to help - I just can't go any faster.
Bob
At 14:43 06/03/2015, Bob Briscoe wrote:
HTTP/2 folks,
As I said, consider this as a late review from a clueful but fresh pair of eyes.
My main concerns with the draft are:
* extensibility (previous posting)
* flow control (this posting - apologies for the length - I've tried to explain properly)
* numerous open issues left dangling (see subsequent postings)
===HTTP/2 FLOW CONTROL===
The term 'window' as used throughout is incorrect and highly confusing, in:
* 'flow control window' (44 occurrences),
* 'initial window size' (5),
* or just 'window size' (8)
* and the terms WINDOW_UPDATE and SETTINGS_INITIAL_WINDOW.
The HTTP/2 WINDOW_UPDATE mechanism constrains HTTP/2 to use only credit-based flow control, not window-based. At one point, it actually says it is credit-based (in flow control principle #2 < https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-5.2.1 > ), but otherwise it incorrectly uses the term window.
This is not just an issue of terminology. The more I re-read the flow control sections the more I became convinced that this terminology is not just /confusing/, rather it's evidence of /confusion/. It raises the questions
* "Is HTTP/2 capable of the flow control it says it's capable of?"
* "What type of flow-control protocol ought HTTP/2 to be capable of?"
* "Can the WINDOW_UPDATE frame support the flow-control that HTTP/2 needs?"
To address these questions, it may help if I separate the two different cases HTTP/2 flow control attempts to cover (my own separation, not from the draft):
a) Intermediate buffer control
Here, a stream's flow enters /and/ leaves a buffer (e.g. at the app-layer of an intermediate node).
b) Flow control by the ultimate client app.
Here flow never releases memory (at least not during the life of the connection). The flow is solely consuming more and more memory (e.g. data being rendered into a client app's memory).
==a) Intermediate buffer control==
For this, sliding window-based flow control would be appropriate, because the goal is to keep the e2e pipeline full without wasting buffer.
Let me prove HTTP/2 cannot do window flow control. For window flow control, the sender needs to be able to advance both the leading and trailing edges of the window. In the draft:
* WINDOW_UPDATE frames can only advance the leading edge of a 'window' (and they are constrained to positive values).
* To advance the trailing edge, window flow control would need a continuous stream of acknowledgements back to the sender (like TCP). The draft does not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at the transport layer, so the sending app-layer cannot advance the trailing edge of a 'window'.
So the protocol can only support credit-based flow control. It is incapable of supporting window flow control.
Next, I don't understand how a receiver can set the credit in 'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to answer the question "How much more can I send than I have seen ACK'd?" that would be easy. But because the protocol is restricted to credit, the sender needs the receiver to answer the much harder open-ended question, "How much more can I send?" So the sender needs the receiver to know how many ACKs the sender has seen, but neither of them know that.
The receiver can try, by taking a guess at the bandwidth-delay product, and adjusting the guess up or down, depending on whether its buffer is growing or shrinking. But this only works if the unknown bandwidth-delay product stays constant.
However, BDP will usually be highly variable, as other streams come and go. So, in the time it takes to get a good estimate of the per-stream BDP, it will probably have changed radically, or the stream will most likely have finished anyway. This is why TCP bases flow control on a window, not credit. By complementing window updates with ACK stream info, a TCP sender has sufficient info to control the flow.
The draft is indeed correct when it says:
"   this can lead to suboptimal use of available
   network resources if flow control is enabled without knowledge of the
   bandwidth-delay product (see [RFC7323]).
"
Was this meant to be a veiled criticism of the protocol's own design? A credit-based flow control protocol like that in the draft does not provide sufficient information for either end to estimate the bandwidth-delay product, given it will be varying rapidly.
==b) Control by the ultimate client app==
For this case, I believe neither window nor credit-based flow control is appropriate:
* There is no memory management issue at the client end - even if there's a separate HTTP/2 layer of memory between TCP and the app, it would be pointless to limit the memory used by HTTP/2, because the data is still going to sit in the same user-space memory (or at least about the same amount of memory) when HTTP/2 passes it over for rendering.
* Nonetheless, the receiving client does need to send messages to the sender to supplement stream priorities, by notifying when the state of the receiving application has changed (e.g. if the user's focus switches from one browser tab to another).
* However, credit-based flow control would be very sluggish for such control, because credit cannot be taken back once it has been given (except HTTP/2 allows SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure that hits all streams together).
==Flow control problem summary==
With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.
IOW: Between a rock and a hard place,... but don't tell them where the rock is.
==Towards a solution?==
I think 'type-a' flow control (for intermediate buffer control) does not need to be at stream-granularity. Indeed, I suspect a proxy could control its app-layer buffering by controlling the receive window of the incoming TCP connection. Has anyone assessed whether this would be sufficient?
I can understand the need for 'type-b' per-stream flow control (by the ultimate client endpoint). Perhaps it would be useful for the receiver to emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream PRIORITY would be sufficient? Either would minimise the response time to a half round trip. Whereas credit flow-control will be much more sluggish (see 'Flow control problem summary').
Either approach would correctly propagate e2e. An intermediate node would naturally tend to prioritise incoming streams that fed into prioritised outgoing streams, so priority updates would tend to propagate from the ultimate receiver, through intermediate nodes, up to the ultimate sender.
==Flow control coverage==
The draft exempts all TCP payload bytes from flow control except HTTP/2 data frames. No rationale is given for this decision. The draft says it's important to manage per-stream memory, then it exempts all the frame types except data, even tho each byte of a non-data frame consumes no less memory than a byte of a data frame.
What message does this put out? "Flow control is not important for one type of bytes with unlimited total size, but flow control is so important that it has to be mandatory for the other type of bytes."
It is certainly critical that WINDOW_UPDATE messages are not covered by flow control, otherwise there would be a real risk of deadlock. It might be that there are dependencies on other frame types that would lead to a dependency loop and deadlock. It would be good to know what the rationale behind these rules was.
==Theory?==
I am concerned that HTTP/2 flow control may have entered new theoretical territory, without suitable proof of safety. The only reassurance we have is one implementation of a flow control algorithm (SPDY), and the anecdotal non-evidence that no-one using SPDY has noticed a deadlock yet (however, is anyone monitoring for deadlocks?).
Whereas SPDY has been an existence proof that an approach like http/2 'works', so far all the flow control algos have been pretty much identical (I think that's true?). I am concerned that the draft takes the InterWeb into uncharted waters, because it allows unconstrained diversity in flow control algos, which is an untested degree of freedom.
The only constraints the draft sets are:
* per-stream flow control is mandatory
* the only protocol message for flow control algos to use is the WINDOW_UPDATE credit message, which cannot be negative
* no constraints on flow control algorithms.
* and all this must work within the outer flow control constraints of TCP.
Some algos might use priority messages to make flow control assumptions. While other algos might associate PRI and WINDOW_UPDATE with different meanings. What confidence do we have that everyone's optimisation algorithms will interoperate? Do we know there will not be certain types of application where deadlock is likely?
"   When using flow
   control, the receiver MUST read from the TCP receive buffer in a
   timely fashion.  Failure to do so could lead to a deadlock when
   critical frames, such as WINDOW_UPDATE, are not read and acted upon.
"
I've been convinced (offlist) that deadlock will not occur as long as the app consumes data 'greedily' from TCP. That has since been articulated in the above normative text. But how sure can we be that every implementer's different interpretations of 'timely' will still prevent deadlock?
Until a good autotuning algorithm for TCP receive window management was developed, good window management code was nearly non-existent. Managing hundreds of interdependent stream buffers is a much harder problem. But implementers are being allowed to just 'Go forth and innovate'. This might work if everyone copies available open source algo(s). But they might not, and they don't have to.
This all seems like 'flying by the seat of the pants'.
==Mandatory Flow Control? ==
"      3. [...] A sender
       MUST respect flow control limits imposed by a receiver."
This ought to be a 'SHOULD' because it is contradicted later - if settings change.
"   6.  Flow control cannot be disabled."
Also effectively contradicted half a page later:
"   Deployments that do not require this capability can advertise a flow
   control window of the maximum size (2^31-1), and by maintaining this
   window by sending a WINDOW_UPDATE frame when any data is received.
   This effectively disables flow control for that receiver."
And contradicted in the definition of half closed (remote):
"  half closed (remote):
      [...] an endpoint is no longer
      obligated to maintain a receiver flow control window.
"
And contradicted in 8.3. The CONNECT Method, which says:
"  Frame types other than DATA
   or stream management frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY)
   MUST NOT be sent on a connected stream, and MUST be treated as a
   stream error (Section 5.4.2) if received.
"
Why is flow control so important that it's mandatory, but so unimportant that you MUST NOT do it when using TLS e2e?
Going back to the earlier quote about using the max window size, it seems perverse for the spec to require endpoints to go through the motions of flow control, even if they arrange for it to affect nothing, but to still require implementation complexity and bandwidth waste with a load of redundant WINDOW_UPDATE frames.
HTTP is used on a wide range of devices, down to the very small and challenged. HTTP/2 might be desirable in such cases, because of the improved efficiency (e.g. header compression), but in many cases the stream model may not be complex enough to need stream flow control.
So why not make flow control optional on the receiving side, but mandatory to implement on the sending side? Then an implementation could have no machinery for tuning window sizes, but it would respond correctly to those set by the other end, which requires much simpler code.
If a receiving implemention chose not to do stream flow control, it could still control flow at the connection (stream 0) level, or at least at the TCP level.
==Inefficiency?==

5.2. Flow Control
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.
==Flow Control Requirements===
I'm not convinced that clear understanding of flow control requirements has driven flow control design decisions.
The draft states various needs for flow-control without giving me a feel of confidence that it has separated out the different cases, and chosen a protocol suitable for each. I tried to go back to the early draft on flow control requirements < http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01 >, and I was not impressed.
I have quoted below the various sentences in the draft that state what flow control is believed to be for. Below that, I have attempted to crystalize out the different concepts, each of which I have tagged within the quotes.
* 2. HTTP/2 Protocol Overview says
  "Flow control and prioritization ensure that it is possible to efficiently use multiplexed streams. [Y]
   Flow control (Section 5.2) helps to ensure that only data that can be used by a receiver is transmitted. [X]"
* 5.2. Flow Control says:
  "Using streams for multiplexing introduces contention over use of the TCP connection [X], resulting in blocked streams [Z]. A flow control scheme ensures that streams on the same connection do not destructively interfere with each other [Z]."
* 5.2.2. Appropriate Use of Flow Control
"  Flow control is defined to protect endpoints that are operating under
   resource constraints.  For example, a proxy needs to share memory
   between many connections, and also might have a slow upstream
   connection and a fast downstream one [Y].  Flow control addresses cases
   where the receiver is unable to process data on one stream, yet wants
   to continue to process other streams in the same connection [X]."
"  Deployments with constrained resources (for example, memory) can
   employ flow control to limit the amount of memory a peer can consume. [Y]
Each requirement has been tagged as follows:
[X] Notification of the receiver's changing utility for each stream
[Y] Prioritisation of streams due to contention over the streaming capacity available to the whole connection.
[Z] Ensuring one stream is not blocked by another.
[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y] sounds more like optimisation across a continuous spectrum.
Regards

Bob
________________________________________________________________
Bob Briscoe,                                                  BT
________________________________________________________________
Bob Briscoe,                                                  BT
________________________________________________________________
Bob Briscoe,                                                  BT

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

________________________________________________________________
Bob Briscoe,                                                  BT

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

________________________________________________________________
Bob Briscoe,                                                  BT


Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Bob Briscoe
In reply to this post by Stuart Douglas
Stuart,

At 00:18 20/03/2015, Stuart Douglas wrote:


==a) Intermediate buffer control==
For this, sliding window-based flow control would be appropriate, because the goal is to keep the e2e pipeline full without wasting buffer.

Let me prove HTTP/2 cannot do window flow control. For window flow control, the sender needs to be able to advance both the leading and trailing edges of the window. In the draft:
* WINDOW_UPDATE frames can only advance the leading edge of a 'window' (and they are constrained to positive values).
* To advance the trailing edge, window flow control would need a continuous stream of acknowledgements back to the sender (like TCP). The draft does not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at the transport layer, so the sending app-layer cannot advance the trailing edge of a 'window'.

So the protocol can only support credit-based flow control. It is incapable of supporting window flow control.


Next, I don't understand how a receiver can set the credit in 'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to answer the question "How much more can I send than I have seen ACK'd?" that would be easy. But because the protocol is restricted to credit, the sender needs the receiver to answer the much harder open-ended question, "How much more can I send?" So the sender needs the receiver to know how many ACKs the sender has seen, but neither of them know that.

The receiver can try, by taking a guess at the bandwidth-delay product, and adjusting the guess up or down, depending on whether its buffer is growing or shrinking. But this only works if the unknown bandwidth-delay product stays constant.

However, BDP will usually be highly variable, as other streams come and go. So, in the time it takes to get a good estimate of the per-stream BDP, it will probably have changed radically, or the stream will most likely have finished anyway. This is why TCP bases flow control on a window, not credit. By complementing window updates with ACK stream info, a TCP sender has sufficient info to control the flow.

The draft is indeed correct when it says:
"   this can lead to suboptimal use of available
   network resources if flow control is enabled without knowledge of the
   bandwidth-delay product (see [RFC7323]).
"
Was this meant to be a veiled criticism of the protocol's own design? A credit-based flow control protocol like that in the draft does not provide sufficient information for either end to estimate the bandwidth-delay product, given it will be varying rapidly.


From my point of view as a server/proxy implementor the flow control window mostly represent the amount of data I am prepared to buffer.Â

From the server as a receiver case we are basically acting as an intermediary between the network and an end user application.

Yes.

In this case the flow control credit basically represents the maximum amount of data that I am prepared to buffer at the HTTP layer. In this case the answer to 'how much more can I send' is simple, it is basically the amount of data I am prepared to buffer. Because I have no idea how quickly (if at all) the user application will consume the data, all I can really do is buffer it and deliver it as the user application requests it.Â

If I set the flow control window to larger than I am prepared to buffer and the application does not consume data quickly enough then I have two options, which are basically stop reading (head of line blocking), or reset the stream, neither of which is particularly good (head of line blocking is particularly problematic if the user application is trying to write a response before reading the request, as window updates will not be processed).Â

If I am only prepared to buffer a small amount of data then my performance is not going to be great no matter what flow control implementation is in use, and I think that this is basically a limitation of a multiplexed protocol (unless you are prepared to accept potential HOL blocking).

This mostly only affects server uploads, as clients will likely have to buffer the whole response anyway so are not constrained by buffer size.Â

All the issues above basically apply to the intermediary/proxy use case as well.Â

Basically I guess what I am getting to is that yes, there are some situations where HTTP2 might perform worse than HTTP1, however I think the underlying problem is intrinsic to any sort of multiplexed protocol, rather than HTTP2's flow control mechanism.

There I disagree. The problem only applies to h2 because it sits above TCP, and is not able to see the ACK information that TCP sees. If stream flow control is implemented in the same layer as both connection flow control and segment acknowledgement, then I suspect it could be made to work properly.

SCTP, Minion and QUIC have this architecture, so implementing stream flow control would not be an unsolvable problem (if a use for it were found, and if it could be made deadlock-free).


==b) Control by the ultimate client app==
For this case, I believe neither window nor credit-based flow control is appropriate:
* There is no memory management issue at the client end - even if there's a separate HTTP/2 layer of memory between TCP and the app, it would be pointless to limit the memory used by HTTP/2, because the data is still going to sit in the same user-space memory (or at least about the same amount of memory) when HTTP/2 passes it over for rendering.


Not necessarily, not all clients are browsers, and even with browsers when downloading a file I imagine the data will generally be transferred straight to disk rather staying in memory. In general I agree with you though, and I think most clients will want to set a large window size.
Â
* Nonetheless, the receiving client does need to send messages to the sender to supplement stream priorities, by notifying when the state of the receiving application has changed (e.g. if the user's focus switches from one browser tab to another).
* However, credit-based flow control would be very sluggish for such control, because credit cannot be taken back once it has been given (except HTTP/2 allows SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure that hits all streams together).


This support is provided by PRIORITY frames, 

Yes (as you can see, I suggested priorities myself). I was assuming per-stream flow control had been proposed for some reason, and having knocked down the other reasons I could think of, the only one left was for the client to use flow control to complement priorities.

You're saying per-stream flow control is not even safe for this case, and your reasoning below seems sound. So I guess you're implying there is very little, if anything, that per-stream flow control is good for. Am I putting words in your mouth?

using flow control for such use cases is very problematic and has a good chance of leading to deadlocks.

Setting aside whether stream flow control is useful,...
... if proper stream flow control were feasible (e.g. if it were implemented in the transport layer), I believe the protocol could be made structurally impossible to deadlock within its own layer (aside from any deadlock at the app-layer, as you point out later). I've written the rules for avoiding flow control deadlock into the Inner Space spec (using understanding gained from multipath TCP). My ambition is to give a proof that it cannot deadlock, even when combined with the byzantine behaviours of middleboxes. It's a tough ambition, but needed to determine any necessary conditions for the proof to hold.

For example consider a Java Servlet container, new requests will be read from the underlying connection, and then dispatched to a worker thread to generate the page. Once a server has started processing a request there is no way to pause the request in a way that frees up the resources (threads, database connections etc) in use. I think almost every use case more complex than simple file serving has this issue, once a server has started processing a request there is generally no way to suspend it and free the resources.Â

I suspect you are right. The deadlock avoidance I was talking about above is a simpler problem (tho by no means trivial), because it solely concerns flow-through buffers, not endpoint resources.


So in your example if the user switches tabs and you stop sending window updates for streams in use by the old browser tab, while sending new requests for the new tab it is possible the server will be in a situation where it is not prepared to allocate resources for the new requests until the existing ones are complete, however these will never complete as the browser has stopped sending window updates.Â

You could argue that a server should limit the max streams value to the number of streams that it is prepared to allocate resources for, however this greatly limits the utility of the priority mechanism, as it means that servers will always handle requests on a first come first served basis. If we set the maximum streams value to a higher amount than what we are prepared to allocate resources for and queue requests then when a request finishes we can pick the highest priority request from the queue to allocate resources to (not to mention there is no round trip delay because the request is queued).

Basically IMHO flow control should not be used to control priority, that is what PRIORITY frames are for, and in general servers can only ever do priority on a best effort approach anyway. If you try and use flow control to enforce a strict priority mechanism you run a very real risk of deadlocks.

That sounds like a reasonable rule, at least for now until we gain better understanding.

This relates to the part of my review about h2 entering uncharted theoretical territory, by allowing implementers free rein on what they do with the protocol elements, with no guidance or constraints. One end might write code that interprets flow control messages with some priority-related semantics, while the implementation at the other end did not intend that.

Â

==Flow control problem summary==

With only a credit signal in the protocol, a receiver is going to have to allow generous credit in the WINDOW_UPDATEs so as not to hurt performance. But then, the receiver will not be able to quickly close down one stream (e.g. when the user's focus changes), because it cannot claw back the generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock is.


For the reasons I outlined above I don't think this is actually a problem.

Certainly, if flow control is not used to close down a stream, then not being able to close down a stream won't be a problem. That does leave the problem of what per-stream flow control /can/ be used for, if it can't be used to slow down streams!

The only believeable use-case I've seen so far is blocking a stream's progress when it is first created.

Also in terms of clawing back credit I thought that in general it was a bad idea? The TCP RFC explicitly states that "shrinking the window" is strongly discouraged, although I must admit I am not fully aware of the reasoning.

From memory, the reasoning was simply that TCP can't unsend what it has already sent. So shrinking the window leaves the protocol in an indeterminate state where more buffer might be needed anyway.


Bob


Stuart
Â

==Towards a solution?==

I think 'type-a' flow control (for intermediate buffer control) does not need to be at stream-granularity. Indeed, I suspect a proxy could control its app-layer buffering by controlling the receive window of the incoming TCP connection. Has anyone assessed whether this would be sufficient?

I can understand the need for 'type-b' per-stream flow control (by the ultimate client endpoint). Perhaps it would be useful for the receiver to emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream PRIORITY would be sufficient? Either would minimise the response time to a half round trip. Whereas credit flow-control will be much more sluggish (see 'Flow control problem summary').

Either approach would correctly propagate e2e. An intermediate node would naturally tend to prioritise incoming streams that fed into prioritised outgoing streams, so priority updates would tend to propagate from the ultimate receiver, through intermediate nodes, up to the ultimate sender.

==Flow control coverage==

The draft exempts all TCP payload bytes from flow control except HTTP/2 data frames. No rationale is given for this decision. The draft says it's important to manage per-stream memory, then it exempts all the frame types except data, even tho each byte of a non-data frame consumes no less memory than a byte of a data frame.

What message does this put out? "Flow control is not important for one type of bytes with unlimited total size, but flow control is so important that it has to be mandatory for the other type of bytes."

It is certainly critical that WINDOW_UPDATE messages are not covered by flow control, otherwise there would be a real risk of deadlock. It might be that there are dependencies on other frame types that would lead to a dependency loop and deadlock. It would be good to know what the rationale behind these rules was.


I think a lot of people had similar concerns. There is a discussion about it here:Â https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0647.html
Â

==Theory?==

I am concerned that HTTP/2 flow control may have entered new theoretical territory, without suitable proof of safety. The only reassurance we have is one implementation of a flow control algorithm (SPDY), and the anecdotal non-evidence that no-one using SPDY has noticed a deadlock yet (however, is anyone monitoring for deadlocks?).

Whereas SPDY has been an existence proof that an approach like http/2 'works', so far all the flow control algos have been pretty much identical (I think that's true?). I am concerned that the draft takes the InterWeb into uncharted waters, because it allows unconstrained diversity in flow control algos, which is an untested degree of freedom.

The only constraints the draft sets are:
* per-stream flow control is mandatory
* the only protocol message for flow control algos to use is the WINDOW_UPDATE credit message, which cannot be negative
* no constraints on flow control algorithms.
* and all this must work within the outer flow control constraints of TCP.

Some algos might use priority messages to make flow control assumptions. While other algos might associate PRI and WINDOW_UPDATE with different meanings. What confidence do we have that everyone's optimisation algorithms will interoperate? Do we know there will not be certain types of application where deadlock is likely?

"   When using flow
   control, the receiver MUST read from the TCP receive buffer in a
   timely fashion.  Failure to do so could lead to a deadlock when
   critical frames, such as WINDOW_UPDATE, are not read and acted upon.
"
I've been convinced (offlist) that deadlock will not occur as long as the app consumes data 'greedily' from TCP. That has since been articulated in the above normative text. But how sure can we be that every implementer's different interpretations of 'timely' will still prevent deadlock?

Until a good autotuning algorithm for TCP receive window management was developed, good window management code was nearly non-existent. Managing hundreds of interdependent stream buffers is a much harder problem. But implementers are being allowed to just 'Go forth and innovate'. This might work if everyone copies available open source algo(s). But they might not, and they don't have to.

This all seems like 'flying by the seat of the pants'.

==Mandatory Flow Control? ==

"      3. [...] A sender
       MUST respect flow control limits imposed by a receiver."
This ought to be a 'SHOULD' because it is contradicted later - if settings change.

"   6.  Flow control cannot be disabled."
Also effectively contradicted half a page later:
"   Deployments that do not require this capability can advertise a flow
   control window of the maximum size (2^31-1), and by maintaining this
   window by sending a WINDOW_UPDATE frame when any data is received.
   This effectively disables flow control for that receiver."

And contradicted in the definition of half closed (remote):
"  half closed (remote):
      [...] an endpoint is no longer
      obligated to maintain a receiver flow control window.
"

And contradicted in 8.3. The CONNECT Method, which says:
"  Frame types other than DATA
   or stream management frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY)
   MUST NOT be sent on a connected stream, and MUST be treated as a
   stream error (Section 5.4.2) if received.
"

Why is flow control so important that it's mandatory, but so unimportant that you MUST NOT do it when using TLS e2e?

Going back to the earlier quote about using the max window size, it seems perverse for the spec to require endpoints to go through the motions of flow control, even if they arrange for it to affect nothing, but to still require implementation complexity and bandwidth waste with a load of redundant WINDOW_UPDATE frames.

HTTP is used on a wide range of devices, down to the very small and challenged. HTTP/2 might be desirable in such cases, because of the improved efficiency (e.g. header compression), but in many cases the stream model may not be complex enough to need stream flow control.

So why not make flow control optional on the receiving side, but mandatory to implement on the sending side? Then an implementation could have no machinery for tuning window sizes, but it would respond correctly to those set by the other end, which requires much simpler code.

If a receiving implemention chose not to do stream flow control, it could still control flow at the connection (stream 0) level, or at least at the TCP level.


==Inefficiency?==

5.2. Flow Control
"Flow control is used for both individual
   streams and for the connection as a whole."
Does this means that every WINDOW_UPDATE on a stream has to be accompanied by another WINDOW_UPDATE frame on stream zero? If so, this seems like 100% message redundancy. Surely I must  have misunderstood.

==Flow Control Requirements===

I'm not convinced that clear understanding of flow control requirements has driven flow control design decisions.

The draft states various needs for flow-control without giving me a feel of confidence that it has separated out the different cases, and chosen a protocol suitable for each. I tried to go back to the early draft on flow control requirements < http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01 >, and I was not impressed.

I have quoted below the various sentences in the draft that state what flow control is believed to be for. Below that, I have attempted to crystalize out the different concepts, each of which I have tagged within the quotes.

* 2. HTTP/2 Protocol Overview says
  "Flow control and prioritization ensure that it is possible to efficiently use multiplexed streams. [Y]
   Flow control (Section 5.2) helps to ensure that only data that can be used by a receiver is transmitted. [X]"
* 5.2. Flow Control says:
  "Using streams for multiplexing introduces contention over use of the TCP connection [X], resulting in blocked streams [Z]. A flow control scheme ensures that streams on the same connection do not destructively interfere with each other [Z]."
* 5.2.2. Appropriate Use of Flow Control
"  Flow control is defined to protect endpoints that are operating under
   resource constraints.  For example, a proxy needs to share memory
   between many connections, and also might have a slow upstream
   connection and a fast downstream one [Y].  Flow control addresses cases
   where the receiver is unable to process data on one stream, yet wants
   to continue to process other streams in the same connection [X]."
"  Deployments with constrained resources (for example, memory) can
   employ flow control to limit the amount of memory a peer can consume. [Y]

Each requirement has been tagged as follows:
[X] Notification of the receiver's changing utility for each stream
[Y] Prioritisation of streams due to contention over the streaming capacity available to the whole connection.
[Z] Ensuring one stream is not blocked by another.

[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y] sounds more like optimisation across a continuous spectrum.


Regards



Bob

________________________________________________________________
Bob Briscoe,                                                  BT

________________________________________________________________
Bob Briscoe,                                                  BT

Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Amos Jeffries-2
In reply to this post by Bob Briscoe
On 21/03/2015 5:50 a.m., Bob Briscoe wrote:

> Jason,
>
> At 16:00 19/03/2015, Jason Greene wrote:
>> I think thats a good argument for why it’s not suitable for
>> rate-limiting relative to a variable bandwidth product of a single
>> connection (which i took as your use-case a).
>
> I believe there is no need for an intermediate node to do flow control
> for individual streams. It does need to control the whole envelope
> within which all the streams are flowing through the proxy's app-layer
> buffer memory (e.g. due to a thick incoming pipe feeding a thin outgoing
> pipe). The best mechanism for controlling the app-layer buffer
> consumption of the aggregate connection is for the intermediate node to
> control the TCP receive window of the incoming stream.
>
> That doesn't preclude the intermediate node passing on any per-stream
> flow control messages emanating from the ultimate receiver so that the
> ultimate sender controls each stream's rate, which will alter the
> balance between streams within the overall envelope at the proxy.
>
> I explained all this in my review.

You seem to not be considering the common case where multiple incoming
pipes potentially match or exceed the size of the pipe they are being
multiplexed to.

Passing the WINDOW_UPDATE from clients to server pipes as-is has great
potential for HOL blocking at the server if any one of the clients
requests a large object with large window size.

Assiging a dedicated Nth of the server pipe to each client leads to
under-utilization if any of the clients pauses. And is very difficult to
guage in advance of N clients existing.

These also happen in reverse when one client fetches from N servers.

And can happen simultaneously in both directions in a multiplexed mesh
of connections. So that server X flooding its connection to client B can
fill client B's pipe and prevent server Y being able to deliver some
frames for client-B which HOL-block server Y from sending client A's
response.


The HTTP/2 scheme allows the intermediary to dynamically offer or
withhold window space/credits to prevent any given stream causing those
problems.


Amos

Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

Roland Zink
In reply to this post by Bob Briscoe
On 20.03.2015 17:50, Bob Briscoe wrote:
> I believe there is no need for an intermediate node to do flow control
> for individual streams. It does need to control the whole envelope
> within which all the streams are flowing through the proxy's app-layer
> buffer memory (e.g. due to a thick incoming pipe feeding a thin
> outgoing pipe). The best mechanism for controlling the app-layer
> buffer consumption of the aggregate connection is for the intermediate
> node to control the TCP receive window of the incoming stream.
>
I think a HTTP proxy can be much more powerful than just forwarding
streams from the server. It may return some stream from cache, answer
some requests directly and forward others, it may even push some
resources for example from cache. As such I don't see a difference to
what an HTTP server may want to do.

> That doesn't preclude the intermediate node passing on any per-stream
> flow control messages emanating from the ultimate receiver so that the
> ultimate sender controls each stream's rate, which will alter the
> balance between streams within the overall envelope at the proxy.
>
> Bob

Regards,
Roland

12