How to handle errors if contents of response are dynamically generated?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

How to handle errors if contents of response are dynamically generated?

X. Long
Hi, I am developing a HTTP server and most of requests the server gets are for large size contents that are dynamically generated.
 
Currently, the server is responsing in the following way: 1) upon receiving a request for object A, it constructs and sends a resposne header with status code 200 (of course no content-length specified) if the object is valid; 2) generates and buffers contents and sends buffered contents by HTTP chunk transfering.
 
The problem here is how to handle potential errors during contents generation in the server. Since a 200 response has already been sent, I cannot send another response with another status code.
 
I can (i) close the connection but the clients (e.g., IE) will just end as the entire data have been received. (2) send nothing and wish clients will timeout themselves. But it seems dangurous that there are chances some clients will be hanging there for a very long time.
 
I am wondering what the "standard" way the server is expected to behavior under such situation. Any comments are welcomed and appreciated!


Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.
Reply | Threaded
Open this post in threaded view
|

Re: How to handle errors if contents of response are dynamically generated?

Mark Baker

This is a well known problem with HTTP.

You should probably ask yourself what it makes sense for clients to do
in case of an error in your server/content, and then try to figure out
how to induce that behaviour.  If you could provide more detail about
your application/content, I might be able to give more specific
advice.

Mark.

On 9/12/07, X. Long <[hidden email]> wrote:

> Hi, I am developing a HTTP server and most of requests the server gets are
> for large size contents that are dynamically generated.
>
> Currently, the server is responsing in the following way: 1) upon receiving
> a request for object A, it constructs and sends a resposne header with
> status code 200 (of course no content-length specified) if the object is
> valid; 2) generates and buffers contents and sends buffered contents by HTTP
> chunk transfering.
>
> The problem here is how to handle potential errors during contents
> generation in the server. Since a 200 response has already been sent, I
> cannot send another response with another status code.
>
> I can (i) close the connection but the clients (e.g., IE) will just end as
> the entire data have been received. (2) send nothing and wish clients will
> timeout themselves. But it seems dangurous that there are chances some
> clients will be hanging there for a very long time.
>
> I am wondering what the "standard" way the server is expected to behavior
> under such situation. Any comments are welcomed and appreciated!
>
> ________________________________
> Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.
>
>


--
Mark Baker.  Ottawa, Ontario, CANADA.         http://www.markbaker.ca
Coactus; Web-inspired integration strategies  http://www.coactus.com

Reply | Threaded
Open this post in threaded view
|

Re: How to handle errors if contents of response are dynamically generated?

X. Long
Thanks Mark! The HTTP server I am building is a server that provide dynamic data (format can be binary or text, size can vary from MB to TB) according to requests of users. The server is supposed to serve user scripts (e.g., in perl/python/java/c#) and browsers (e.g., ie).
 
The expected solution would be the client can get aware of the error immediately (not waiting for timeout and knowing the data is incomplete). Ideally, users should be notified this is an internal error and they should contact admin instead of repeating the request.
 
Also, since this is a well known problem with HTTP, what are the general approaches recommended? It will be great if some previous discussions about this topic can be pointed to me.
 
Again, thanks a lot!

Mark Baker <[hidden email]> wrote:
This is a well known problem with HTTP.

You should probably ask yourself what it makes sense for clients to do
in case of an error in your server/content, and then try to figure out
how to induce that behaviour. If you could provide more detail about
your application/content, I might be able to give more specific
advice.

Mark.

On 9/12/07, X. Long wrote:

> Hi, I am developing a HTTP server and most of requests the server gets are
> for large size contents that are dynamically generated.
>
> Currently, the server is responsing in the following way: 1) upon receiving
> a request for object A, it constructs and sends a resposne header with
> status code 200 (of course no content-length specified) if the object is
> valid; 2) generates and buffers contents and sends buffered contents by HTTP
> chunk transfering.
>
> The problem here is how to handle potential errors during contents
> generation in the server. Since a 200 response has already been sent, I
> cannot send another response with another status code.
>
> I can (i) close the connection but the clients (e.g., IE) will just end as
> the entire data have been received. (2) send nothing and wish clients will
> timeout themselves. But it seems dangurous that there are chances some
> clients will be hanging there for a very long time.
>
> I am wondering what the "standard" way the server is expected to behavior
> under such situation. Any comments are welcomed and appreciated!
>
> ________________________________
> Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.
>
>


--
Mark Baker. Ottawa, Ontario, CANADA. http://www.markbaker.ca
Coactus; Web-inspired integration strategies http://www.coactus.com


Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.
Reply | Threaded
Open this post in threaded view
|

Re: How to handle errors if contents of response are dynamically generated?

Mark Baker

On 9/13/07, X. Long <[hidden email]> wrote:
> Thanks Mark! The HTTP server I am building is a server that provide dynamic
> data (format can be binary or text, size can vary from MB to TB) according
> to requests of users. The server is supposed to serve user scripts (e.g., in
> perl/python/java/c#) and browsers (e.g., ie).

Hmm, that kind of generality means you need a very general solution.
I'm not aware of a single solution which would meet your needs.

Had you said that you were using just one or two specific data
formats, then you might consider injecting an extension into the data
that signalled the error (if the format supported that, plus with some
script that could render it in a browser).  That doesn't sound general
enough for you though.

> The expected solution would be the client can get aware of the error
> immediately (not waiting for timeout and knowing the data is incomplete).
> Ideally, users should be notified this is an internal error and they should
> contact admin instead of repeating the request.
>
> Also, since this is a well known problem with HTTP, what are the general
> approaches recommended? It will be great if some previous discussions about
> this topic can be pointed to me.

It's well known, but not so oft-encountered AFAIK.  You might try
digging through the IETF HTTP WG archives at
http://lists.w3.org/Archives/Public/ietf-http-wg/

Sorry I couldn't be more help.

Mark.
--
Mark Baker.  Ottawa, Ontario, CANADA.         http://www.markbaker.ca
Coactus; Web-inspired integration strategies  http://www.coactus.com

Reply | Threaded
Open this post in threaded view
|

RE: How to handle errors if contents of response are dynamically generated?

Mike Dierken

How much data is buffered on the server?
If that dynamically generated can be stored (at least temporarily), a root
document with links to chunks/fragments can be returned quickly and each
chuck would be a separate request, with it's own response status code.


> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]] On Behalf Of Mark Baker
> Sent: Thursday, September 13, 2007 8:38 PM
> To: X. Long
> Cc: [hidden email]
> Subject: Re: How to handle errors if contents of response are
> dynamically generated?
>
>
> On 9/13/07, X. Long <[hidden email]> wrote:
> > Thanks Mark! The HTTP server I am building is a server that provide
> > dynamic data (format can be binary or text, size can vary
> from MB to
> > TB) according to requests of users. The server is supposed to serve
> > user scripts (e.g., in
> > perl/python/java/c#) and browsers (e.g., ie).
>
> Hmm, that kind of generality means you need a very general solution.
> I'm not aware of a single solution which would meet your needs.
>
> Had you said that you were using just one or two specific
> data formats, then you might consider injecting an extension
> into the data that signalled the error (if the format
> supported that, plus with some script that could render it in
> a browser).  That doesn't sound general enough for you though.
>
> > The expected solution would be the client can get aware of
> the error
> > immediately (not waiting for timeout and knowing the data
> is incomplete).
> > Ideally, users should be notified this is an internal error
> and they
> > should contact admin instead of repeating the request.
> >
> > Also, since this is a well known problem with HTTP, what are the
> > general approaches recommended? It will be great if some previous
> > discussions about this topic can be pointed to me.
>
> It's well known, but not so oft-encountered AFAIK.  You might
> try digging through the IETF HTTP WG archives at
> http://lists.w3.org/Archives/Public/ietf-http-wg/
>
> Sorry I couldn't be more help.
>
> Mark.
> --
> Mark Baker.  Ottawa, Ontario, CANADA.         http://www.markbaker.ca
> Coactus; Web-inspired integration strategies  http://www.coactus.com
>


Reply | Threaded
Open this post in threaded view
|

RE: How to handle errors if contents of response are dynamically generated?

Greg Robson-Garth
In reply to this post by X. Long

Hi

 

There is no standard way of handling this situation. Once you have committed to sending the data, and your server subsequently encounters a problem, then you have very little option but to close the connection. You cannot assume the client will ever timeout and you cannot really be accountable for what

 

According to the specification, the OPTIONAL chunked trailer… (3.6.1) “allows dynamically produced content to be transferred along with information necessary for the recipient to verify that it has received the full message”

 

If you have some influence over the way the client processes the messages, then you could do a number of things, for example

 

-          Send a Content-Length trailer (ref issue in 4.4-3)

-          Send an X- header trailer indicating a closure without completion

 

One would have thought that a -1 (or some other indicator value for example) instead of chunked size could have been used together with some additional text to indicate a failure.

 

--Greg

 


From: [hidden email] [mailto:[hidden email]] On Behalf Of X. Long
Sent: Friday, 14 September 2007 5:11 AM
To: Mark Baker
Cc: [hidden email]
Subject: Re: How to handle errors if contents of response are dynamically generated?

 

Thanks Mark! The HTTP server I am building is a server that provide dynamic data (format can be binary or text, size can vary from MB to TB) according to requests of users. The server is supposed to serve user scripts (e.g., in perl/python/java/c#) and browsers (e.g., ie).

 

The expected solution would be the client can get aware of the error immediately (not waiting for timeout and knowing the data is incomplete). Ideally, users should be notified this is an internal error and they should contact admin instead of repeating the request.

 

Also, since this is a well known problem with HTTP, what are the general approaches recommended? It will be great if some previous discussions about this topic can be pointed to me.

 

Again, thanks a lot!


Mark Baker <[hidden email]> wrote:

This is a well known problem with HTTP.

You should probably ask yourself what it makes sense for clients to do
in case of an error in your server/content, and then try to figure out
how to induce that behaviour. If you could provide more detail about
your application/content, I might be able to give more specific
advice.

Mark.

On 9/12/07, X. Long wrote:
> Hi, I am developing a HTTP server and most of requests the server gets are
> for large size contents that are dynamically generated.
>
> Currently, the server is responsing in the following way: 1) upon receiving
> a request for object A, it constructs and sends a resposne header with
> status code 200 (of course no content-length specified) if the object is
> valid; 2) generates and buffers contents and sends buffered contents by HTTP
> chunk transfering.
>
> The problem here is how to handle potential errors during contents
> generation in the server. Since a 200 response has already been sent, I
> cannot send another response with another status code.
>
> I can (i) close the connection but the clients (e.g., IE) will just end as
> the entire data have been received. (2) send nothing and wish clients will
> timeout themselves. But it seems dangurous that there are chances some
> clients will be hanging there for a very long time.
>
> I am wondering what the "standard" way the server is expected to behavior
> under such situation. Any comments are welcomed and appreciated!
>
> ________________________________
> Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.
>
>


--
Mark Baker. Ottawa, Ontario, CANADA. http://www.markbaker.ca
Coactus; Web-inspired integration strategies http://www.coactus.com

 

 


Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.

Reply | Threaded
Open this post in threaded view
|

RE: How to handle errors if contents of response are dynamically generated?

X. Long
In reply to this post by Mike Dierken
We cannot buffer data on the server because the data can be very large.
 
Thank everyone for your suggestions and helps!

Mike Dierken <[hidden email]> wrote:
How much data is buffered on the server?
If that dynamically generated can be stored (at least temporarily), a root
document with links to chunks/fragments can be returned quickly and each
chuck would be a separate request, with it's own response status code.


> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]] On Behalf Of Mark Baker
> Sent: Thursday, September 13, 2007 8:38 PM
> To: X. Long
> Cc: [hidden email]
> Subject: Re: How to handle errors if contents of response are
> dynamically generated?
>
>
> On 9/13/07, X. Long wrote:
> > Thanks Mark! The HTTP server I am building is a server that provide
> > dynamic data (format can be binary or text, size can vary
> from MB to
> > TB) according to requests of users. The server is supposed to serve
> > user scripts (e.g., in
> > perl/python/java/c#) and browsers (e.g., ie).
>
> Hmm, that kind of generality means you need a very general solution.
> I'm not aware of a single solution which would meet your needs.
>
> Had you said that you were using just one or two specific
> data formats, then you might consider injecting an extension
> into the data that signalled the error (if the format
> supported that, plus with some script that could render it in
> a browser). That doesn't sound general enough for you though.
>
> > The expected solution would be the client can get aware of
> the error
> > immediately (not waiting for timeout and knowing the data
> is incomplete).
> > Ideally, users should be notified this is an internal error
> and they
> > should contact admin instead of repeating the request.
> >
> > Also, since this is a well known problem with HTTP, what are the
> > general approaches recommended? It will be great if some previous
> > discussions about this topic can be pointed to me.
>
> It's well known, but not so oft-encountered AFAIK. You might
> try digging through the IETF HTTP WG archives at
> http://lists.w3.org/Archives/Public/ietf-http-wg/
>
> Sorry I couldn't be more help.
>
> Mark.
> --
> Mark Baker. Ottawa, Ontario, CANADA. http://www.markbaker.ca
> Coactus; Web-inspired integration strategies http://www.coactus.com
>



Be a better Heartthrob. Get better relationship answers from someone who knows.
Yahoo! Answers - Check it out.
Reply | Threaded
Open this post in threaded view
|

RE: How to handle errors if contents of response are dynamically generated?

Mike Dierken
Can you generate a range of data later, when a second or further request arrives for that portion?
 


From: [hidden email] [mailto:[hidden email]] On Behalf Of X. Long
Sent: Friday, September 14, 2007 2:18 PM
To: [hidden email]
Subject: RE: How to handle errors if contents of response are dynamically generated?

We cannot buffer data on the server because the data can be very large.
 
Thank everyone for your suggestions and helps!

Mike Dierken <[hidden email]> wrote:
How much data is buffered on the server?
If that dynamically generated can be stored (at least temporarily), a root
document with links to chunks/fragments can be returned quickly and each
chuck would be a separate request, with it's own response status code.


> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]] On Behalf Of Mark Baker
> Sent: Thursday, September 13, 2007 8:38 PM
> To: X. Long
> Cc: [hidden email]
> Subject: Re: How to handle errors if contents of response are
> dynamically generated?
>
>
> On 9/13/07, X. Long wrote:
> > Thanks Mark! The HTTP server I am building is a server that provide
> > dynamic data (format can be binary or text, size can vary
> from MB to
> > TB) according to requests of users. The server is supposed to serve
> > user scripts (e.g., in
> > perl/python/java/c#) and browsers (e.g., ie).
>
> Hmm, that kind of generality means you need a very general solution.
> I'm not aware of a single solution which would meet your needs.
>
> Had you said that you were using just one or two specific
> data formats, then you might consider injecting an extension
> into the data that signalled the error (if the format
> supported that, plus with some script that could render it in
> a browser). That doesn't sound general enough for you though.
>
> > The expected solution would be the client can get aware of
> the error
> > immediately (not waiting for timeout and knowing the data
> is incomplete).
> > Ideally, users should be notified this is an internal error
> and they
> > should contact admin instead of repeating the request.
> >
> > Also, since this is a well known problem with HTTP, what are the
> > general approaches recommended? It will be great if some previous
> > discussions about this topic can be pointed to me.
>
> It's well known, but not so oft-encountered AFAIK. You might
> try digging through the IETF HTTP WG archives at
> http://lists.w3.org/Archives/Public/ietf-http-wg/
>
> Sorry I couldn't be more help.
>
> Mark.
> --
> Mark Baker. Ottawa, Ontario, CANADA. http://www.markbaker.ca
> Coactus; Web-inspired integration strategies http://www.coactus.com
>



Be a better Heartthrob. Get better relationship answers from someone who knows.
Yahoo! Answers - Check it out.
Reply | Threaded
Open this post in threaded view
|

Re: How to handle errors if contents of response are dynamically generated?

Mark Nottingham-4
In reply to this post by Greg Robson-Garth

If you're using chunked encoding, closing the connection without  
sending the final zero-sized chunk will indicate a problem.


On 2007/09/14, at 5:29 PM, Greg Robson-Garth wrote:

> One would have thought that a -1 (or some other indicator value for  
> example) instead of chunked size could have been used together with  
> some additional text to indicate a failure.
>

--
Mark Nottingham       [hidden email]