exposing CANVAS or something like it to Web Workers

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
41 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Charles Pritchard-2
On 5/14/2012 6:08 PM, Boris Zbarsky wrote:
> On 5/14/12 8:58 PM, Charles Pritchard wrote:
>> I agree... Can we get this off the main thread?
>
> "Maybe".
>
> It would be pretty nontrivial in Gecko; last I looked it would be
> pretty painful in WebKit too.  Can't speak for other UAs.

Can it be pumped through what's essentially an iframe on a null origin?

I don't know enough about browser internals to help on this one.
Yes, loading an SVG image is a heavy call.

For 90% of the SVG content out there, it'd probably be faster to parse
and draw the SVG via Canvas and JS.
Still it's a hell of a lot nicer to load it via <img> tag.

I'd just assumed that <img src="pic.svg" /> was loaded off-thread /
async much like <img> calls.

There's nothing that gets carried from the document to the <img> other
than the width/height, which I believe is carried through.
Which is a good thing, of course.


-Charles

Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Charles Pritchard-2
In reply to this post by Gregg Tavares (wrk)
On 5/14/2012 6:14 PM, Gregg Tavares (勤) wrote:


On Mon, May 14, 2012 at 6:07 PM, Boris Zbarsky <[hidden email]> wrote:
On 5/14/12 8:55 PM, Gregg Tavares (勤) wrote:
   1)  Various canvas 2d context methods depend on the styles of the
   canvas to define how they actually behave.  Clearly this would need
   some sort of changes for Workers anyway; the question is what those
   changes would need to be.

Which methods are these?

Anything involving setting color (e.g. the strokeStyle setter, the fillStyle setter), due to "currentColor".  Anything involving text because font styles come from the element or document.

Good to know.

That doesn't sound like a showstopper though. If a canvas/CanvasSurface is available in workers the simplest solution would just be that "currentColor" defaults to something "black?" or nothing "". Pick one.

Font is still a little tricky from loading fonts via CSS. Font is tricky anyway, though, so it wouldn't be that much of a step backward.

Would we assume that if a font is available from the parent context it's going to be available to the worker?

currentColor would just default to black, as we're not talking about a color inherited from the DOM.


 
Those are the ones that come to mind offhand, but I haven't looked at the various recent additions to the 2d context closely.

The recent additions are more proposals than additions. They're proposals from Tab and Ian and not yet implemented.

They revolve around a Path object, which we've not yet discussed.
Otherwise, they include lightweight nodes and DOM fallback content, which isn't relevant to workers.

My thinking is the same as yours: fillStyle/strokeStyle and font are the ones that come to mind.
Pattern and Gradient are items that could conceivably be cloned and/or shared.
They can both be as efficient as sending Blob via postMessage.

I think Gregg was just settling on not-sending Canvas over postMessage but rather creating the instance inside of each Worker.






Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Boris Zbarsky
In reply to this post by Charles Pritchard-2
On 5/14/12 9:15 PM, Charles Pritchard wrote:

> On 5/14/2012 6:08 PM, Boris Zbarsky wrote:
>> On 5/14/12 8:58 PM, Charles Pritchard wrote:
>>> I agree... Can we get this off the main thread?
>>
>> "Maybe".
>>
>> It would be pretty nontrivial in Gecko; last I looked it would be
>> pretty painful in WebKit too. Can't speak for other UAs.
>
> Can it be pumped through what's essentially an iframe on a null origin?

The issue at least in Gecko is that the DOM uses various
effectively-global variables (shared caches of various sorts,
non-threadsafe services, etc).  The origin is irrelevant.

> I'd just assumed that <img src="pic.svg" /> was loaded off-thread /
> async much like <img> calls.

The _loading_ is async.  In Gecko the parsing of the XML, DOM
construction, style computations, layout are on the main thread, like
they are for HTML.

It's the "DOM construction, style computations, layout" parts that are
not so easy to push off to a background thread...

-Boris

Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Jonas Sicking-2
In reply to this post by Charles Pritchard-2
On Mon, May 14, 2012 at 5:10 PM, Charles Pritchard <[hidden email]> wrote:

> On May 14, 2012, at 4:42 PM, Jonas Sicking <[hidden email]> wrote:
>
>> On Mon, May 14, 2012 at 3:28 PM, Glenn Maynard <[hidden email]> wrote:
>>> On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤) <[hidden email]> wrote:
>>>
>>>> I'd like to work on exposing something like CANVAS to web workers.
>>>>
>>>> Ideally how over it works I'd like to be able to
>>>>
>>>> *) get a 2d context in a web worker
>>>
>>>
>>> I'd recommend not trying to tackle 2d and 3d contexts at once, and only
>>> worrying about WebGL to start.
>>>
>>> Another issue: rendering in a worker thread onto a canvas which is displayed
>>> in the main thread.  This needs to be solved in a way that doesn't cause the
>>> asynchronous nature of what's happening to be visible to scripts.  toDataURL
>>> and toBlob would probably need to be prohibited on the canvas element.  I'm
>>> not sure what the actual API would look like.
>>
>> If/when we do this, I think it should be done in such a way that the
>> main window can't access the canvas object at all. Similar to what
>> happens when an ArrayBuffer is transferred to a Worker using
>> structured cloning. Once a canvas is transferred to a Worker, any
>> access to it should throw or return null/0/"". If you want to transfer
>> pixel data to the main thread, it seems less racy to do that by
>> getting the pixel data in the Worker which owns the canvas and then
>> transfer that to the main thread using postMessage.
>>
>>> This would also require some
>>> equivalent to requestAnimationFrame in the worker thread.
>>
>> Agreed!
>>
>> / Jonas
>
>
>
> I'm a bit lost-- wouldn't we just postMessage from the document over to the web worker when we want a refresh?
>
> I agree that we ought to be transferring pixel data not Canvas contexts; with the possible exception of CSS context.
>
> We could just create new canvas instances inside the worker thread. I'd still prefer to clone CanvasPattern as a means of transferring paint over to the worker, though sending pixel data would work too.
>
> I heard Picture come up-- it seems like that object might have additional semantics for high resolution alternatives that may need to be considered.

I was saying that I think we should transfer the context from the main
thread to the worker thread. The worker thread should then be able to
use the context to draw directly to the screen without interacting
with the main thread.

You should generally not need to transfer pixel data. But I think it
should be possible to grab pixel data on whichever thread is currently
owning a context. This will implicitly make it possible to transfer
pixel data if anyone wants to do it.

/ Jonas

Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Charles Pritchard-2
On 5/14/2012 6:23 PM, Jonas Sicking wrote:

>>>> >>>  This would also require some
>>>> >>>  equivalent to requestAnimationFrame in the worker thread.
>>> >>
>>> >>  Agreed!
>>> >>
>>> >>  / Jonas
>> >
>> >
>> >
>> >  I'm a bit lost-- wouldn't we just postMessage from the document over to the web worker when we want a refresh?
>> >
>> >  I agree that we ought to be transferring pixel data not Canvas contexts; with the possible exception of CSS context.
>> >
>> >  We could just create new canvas instances inside the worker thread. I'd still prefer to clone CanvasPattern as a means of transferring paint over to the worker, though sending pixel data would work too.
>> >
>> >  I heard Picture come up-- it seems like that object might have additional semantics for high resolution alternatives that may need to be considered.
> I was saying that I think we should transfer the context from the main
> thread to the worker thread. The worker thread should then be able to
> use the context to draw directly to the screen without interacting
> with the main thread.
>
> You should generally not need to transfer pixel data. But I think it
> should be possible to grab pixel data on whichever thread is currently
> owning a context. This will implicitly make it possible to transfer
> pixel data if anyone wants to do it.

OK, that's the same concept as I was hoping for with
document.getCSSCanvasContext.

Mozilla is a bit ahead of webkit, going with  "element(any-element)" --
webkit has "-webkit-canvas(css-canvas-id)".

It's a very different approach. I'd still go ahead and just have
requestAnimationFrame pump events from the main frame.
There are so many things that could block the main frame to where an rAF
repaint just isn't necessary from the worker-end.

It may also keep some animations continuing. I'm not positive, but
something like window.prompt may block execution on the main thread, but
if rAF were running on the worker, its repaints would still happen.
Generally, while the main thread is blocked, repainting on it is not
going to be that useful; we're shuffling the repaints off so they aren't
the cause of blocking.

Those are the two sides of it that I see... but it's still a very
different proposal than the idea of just having non-transferable canvas
contexts.

...

Boris brought up some good points about possible blocking in
implementations with loading SVG images in the worker thread via Picture
(or a chopped down Image). At present, those sound more like
implementation-issues, not particularly issues in the validity of a spec.
http://lists.w3.org/Archives/Public/public-webapps/2012AprJun/0744.html

They do sound a little unfortunate, but I'd sooner trade some possible
blocking between a Worker and the main thread, than having to carry a
full SVG parser written in JS in my worker threads.
As an author, I've always had to take precautions for SVG and decide
whether I want to render or give myself up to the browser implementation.

-Charles

Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Gregg Tavares (wrk)
In reply to this post by Jonas Sicking-2


On Mon, May 14, 2012 at 6:23 PM, Jonas Sicking <[hidden email]> wrote:
On Mon, May 14, 2012 at 5:10 PM, Charles Pritchard <[hidden email]> wrote:
> On May 14, 2012, at 4:42 PM, Jonas Sicking <[hidden email]> wrote:
>
>> On Mon, May 14, 2012 at 3:28 PM, Glenn Maynard <[hidden email]> wrote:
>>> On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤) <[hidden email]> wrote:
>>>
>>>> I'd like to work on exposing something like CANVAS to web workers.
>>>>
>>>> Ideally how over it works I'd like to be able to
>>>>
>>>> *) get a 2d context in a web worker
>>>
>>>
>>> I'd recommend not trying to tackle 2d and 3d contexts at once, and only
>>> worrying about WebGL to start.
>>>
>>> Another issue: rendering in a worker thread onto a canvas which is displayed
>>> in the main thread.  This needs to be solved in a way that doesn't cause the
>>> asynchronous nature of what's happening to be visible to scripts.  toDataURL
>>> and toBlob would probably need to be prohibited on the canvas element.  I'm
>>> not sure what the actual API would look like.
>>
>> If/when we do this, I think it should be done in such a way that the
>> main window can't access the canvas object at all. Similar to what
>> happens when an ArrayBuffer is transferred to a Worker using
>> structured cloning. Once a canvas is transferred to a Worker, any
>> access to it should throw or return null/0/"". If you want to transfer
>> pixel data to the main thread, it seems less racy to do that by
>> getting the pixel data in the Worker which owns the canvas and then
>> transfer that to the main thread using postMessage.
>>
>>> This would also require some
>>> equivalent to requestAnimationFrame in the worker thread.
>>
>> Agreed!
>>
>> / Jonas
>
>
>
> I'm a bit lost-- wouldn't we just postMessage from the document over to the web worker when we want a refresh?
>
> I agree that we ought to be transferring pixel data not Canvas contexts; with the possible exception of CSS context.
>
> We could just create new canvas instances inside the worker thread. I'd still prefer to clone CanvasPattern as a means of transferring paint over to the worker, though sending pixel data would work too.
>
> I heard Picture come up-- it seems like that object might have additional semantics for high resolution alternatives that may need to be considered.

I was saying that I think we should transfer the context from the main
thread to the worker thread. The worker thread should then be able to
use the context to draw directly to the screen without interacting
with the main thread.

That's a great goal but is transferring the right way to do it?

context.canvas is currently a reference to the canvas from which the context as created. Will that just go NULL on transfer? Will it change to some other object that is not a Canvas but acts similarly?

What happens to the context on the main page? All of it's methods become no-ops and it's properties become non-live?

How will things synchronize if I call canvas.toDataURL after passing the context to a work? Exception? nothing? It seems hard to define how that would synchronize

If there was a way to allow a worker to generate a media stream and one of those ways involves an in worker canvas/canvassurface, then it seems like none of the above questions have to be answered. Instead the worker creates the surface, and then through some other API attached it to a video tag. You can call video.toDataURL, that's well defined (I assume)

Synchronization is also somewhat defined in that direction. Doesn't that seem like the path of least resistance?



 

You should generally not need to transfer pixel data. But I think it
should be possible to grab pixel data on whichever thread is currently
owning a context. This will implicitly make it possible to transfer
pixel data if anyone wants to do it.

/ Jonas

Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Charles Pritchard-2
In reply to this post by Boris Zbarsky
On 5/14/2012 6:07 PM, Boris Zbarsky wrote:

> On 5/14/12 8:55 PM, Gregg Tavares (勤) wrote:
>>     1)  Various canvas 2d context methods depend on the styles of the
>>     canvas to define how they actually behave.  Clearly this would need
>>     some sort of changes for Workers anyway; the question is what those
>>     changes would need to be.
>>
>> Which methods are these?
>
> Anything involving setting color (e.g. the strokeStyle setter, the
> fillStyle setter), due to "currentColor".  Anything involving text
> because font styles come from the element or document.
>
> Those are the ones that come to mind offhand, but I haven't looked at
> the various recent additions to the 2d context closely.

What would web fonts do in this situation, in Mozilla? If I've confirmed
that a font is loaded in the main thread, would it be available to a
worker for use in rendering?

Some implementations of Canvas font methods have been buggy, but
dropping fillText/strokeText altogether would be a loss and may look
strange in the specs.
I've seen font not working correctly with other canvas state variables
in WebKit + Chrome; so it's not as though font is fully supported in the
main thread, currently.

It may be easier to postpone Image / Picture semantics in workers. I
think patterns could still be salvaged from createPattern.
It sounds like there are practical issues in current implementations.

When authoring,
With a picture, I can still xhr request it in the worker, then I send it
to the main thread, and load it in the main thread, then send it back to
the worker.
As an author, if I'm loading images I expect them to be a normal part of
the page's load time and responsiveness.... I can still off-load my
js-heavy work onto workers.

In Canvas 2d anyway, I can easily get by without picture in workers: (a
= new Picture()).src = ....; a.onload { ctx.drawImage(a,0,0); };  if I
can use fillRect with pattern.
Otherwise, I'd have to do the extra steps of pushing pixel arrays back
and forth.

All of the extra work is still worth it to get work done in workers, in
cases where this level of work is needed. It's just a few extra lines of
JS code (every time).

-Charles




Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Boris Zbarsky
On 5/14/12 10:00 PM, Charles Pritchard wrote:
> What would web fonts do in this situation, in Mozilla?

Probably cry.  ;)

> If I've confirmed that a font is loaded in the main thread, would it be available to a
> worker for use in rendering?

Not without some pretty serious reworking.  Which might need to happen.

Of course basic text layout would also not be available without some
serious reworking (e.g. making the textrun cache threadsafe or creating
per-thread textrun caches or something), so the question of web fonts is
somewhat academic at the moment.

-Boris

Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Charles Pritchard-2
On 5/14/2012 7:09 PM, Boris Zbarsky wrote:

> On 5/14/12 10:00 PM, Charles Pritchard wrote:
>> What would web fonts do in this situation, in Mozilla?
>
> Probably cry.  ;)
>
>> If I've confirmed that a font is loaded in the main thread, would it
>> be available to a
>> worker for use in rendering?
>
> Not without some pretty serious reworking.  Which might need to happen.
>
> Of course basic text layout would also not be available without some
> serious reworking (e.g. making the textrun cache threadsafe or
> creating per-thread textrun caches or something), so the question of
> web fonts is somewhat academic at the moment.
>

I meant solely for Canvas 2d.

I can live with staying away from fillText/strokeText on a worker thread
if I'm loading fonts.
It's been broken on the main thread anyway, requiring intermediate
Canvas surfaces for some operations.

...

SVG image and drawImage is mixed anyway; we can't transfer the data
between threads as drawImage SVG will usually flag the Canvas as dirty
in implementations.

We could just use Canvas 2d to handle pattern uploads for WebGL. Seems
like that'd work without requiring fancy footwork to gain Picture/Image
support in the worker.
SVG images would get fixed some other day.


-Charles







Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Boris Zbarsky
On 5/14/12 10:18 PM, Charles Pritchard wrote:

> On 5/14/2012 7:09 PM, Boris Zbarsky wrote:
>> On 5/14/12 10:00 PM, Charles Pritchard wrote:
>>> What would web fonts do in this situation, in Mozilla?
>>
>> Probably cry. ;)
>>
>>> If I've confirmed that a font is loaded in the main thread, would it
>>> be available to a
>>> worker for use in rendering?
>>
>> Not without some pretty serious reworking. Which might need to happen.
>>
>> Of course basic text layout would also not be available without some
>> serious reworking (e.g. making the textrun cache threadsafe or
>> creating per-thread textrun caches or something), so the question of
>> web fonts is somewhat academic at the moment.
>>
>
> I meant solely for Canvas 2d.

Yes, I understand that.  Canvas 2d text still needs to be able to do
things like font fallback, shaping, bidi, etc, etc, etc. last I checked.

-Boris

Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Charles Pritchard-2
On 5/14/2012 7:24 PM, Boris Zbarsky wrote:

> On 5/14/12 10:18 PM, Charles Pritchard wrote:
>> On 5/14/2012 7:09 PM, Boris Zbarsky wrote:
>>> On 5/14/12 10:00 PM, Charles Pritchard wrote:
>>>> What would web fonts do in this situation, in Mozilla?
>>>
>>> Probably cry. ;)
>>>
>>>> If I've confirmed that a font is loaded in the main thread, would it
>>>> be available to a
>>>> worker for use in rendering?
>>>
>>> Not without some pretty serious reworking. Which might need to happen.
>>>
>>> Of course basic text layout would also not be available without some
>>> serious reworking (e.g. making the textrun cache threadsafe or
>>> creating per-thread textrun caches or something), so the question of
>>> web fonts is somewhat academic at the moment.
>>>
>>
>> I meant solely for Canvas 2d.
>
> Yes, I understand that.  Canvas 2d text still needs to be able to do
> things like font fallback, shaping, bidi, etc, etc, etc. last I checked.

Oh, the rendering isn't thread safe either? Yes, Canvas 2d text does [is
supposed to] use all of those items.

Well, I'll give up strokeText/fillText entirely in workers if it'll get
me the goods faster.
For a11y, I'm going to need to track my text in the main thread anyway.
I can pre-render there if need be.

-Charles

Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Boris Zbarsky
On 5/14/12 10:32 PM, Charles Pritchard wrote:
>> Yes, I understand that. Canvas 2d text still needs to be able to do
>> things like font fallback, shaping, bidi, etc, etc, etc. last I checked.
>
> Oh, the rendering isn't thread safe either?

Right.  The rendering involves access to style data structures that are
not threadsafe.  It involves caching shaped textruns and such; that
cache is not threadsafe.  The actual shaper may or may not be
threadsafe.  The bidi code is not threadsafe.

These are all solvable issues... on some timescale.  ;)

-Boris

Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Gregg Tavares (wrk)
So how to take this forward?

My #1 priority is to get WebGL in workers. Lots of developers have expressed a need for this from decoding compressed textures in a worker to offloading thousands of draw calls per frame to a worker. WebGL already defines sharing mechanisms so at least for the WebGL case I don't need to solve how to display anything back in the main worker. The worker can draw to a texture. The main page can use that texture. WebGL/OpenGL already define that relationship.

But, to get WebGL in a worker I need a way to get a WebGLRenderingContext into a worker. 

Some Ideas

*) Create context in main page, pass to worker

Pros:  

. No new APIs or objects.

Cons:

. Transfer is messy. 

What happens to all the entry points and properties to the context object left in the main page? 
What happens to the "cavnas" parameter on the transferred context?
How do you synchronize the main page setting canvas.width or canvas.height with the worker trying to render to it?


*) Create a context directly in a Worker (or anywhere for that matter)

As in "var gl = new WebGLRenderingContext"

Pros:

. ???

Cons:

. requires defining how the backbuffer size is set.
   Maybe there is no backbuffer for a directly created WebGLRenderingContext?
. if there is no backbuffer then using one of these contexts to draw into a texture
  or 2d context is problematic


*) Create an offscreen canvas like object minus HTMLElement parts

Pros:

. same or nearly the API as already exists for canvas
. flexible. getContext can return different kinds of context. Maybe only "webgl" for now?

Cons:

. ???




Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

妟婹
On Wed, May 16, 2012 at 12:30 PM, Gregg Tavares (勤) <[hidden email]> wrote:

> So how to take this forward?
>
> My #1 priority is to get WebGL in workers. Lots of developers have expressed
> a need for this from decoding compressed textures in a worker to offloading
> thousands of draw calls per frame to a worker. WebGL already defines sharing
> mechanisms so at least for the WebGL case I don't need to solve how to
> display anything back in the main worker. The worker can draw to a texture.
> The main page can use that texture. WebGL/OpenGL already define that
> relationship.
>
> But, to get WebGL in a worker I need a way to get a WebGLRenderingContext
> into a worker.
>
> Some Ideas
>
> *) Create context in main page, pass to worker
>
> Pros:
>
> . No new APIs or objects.
>
> Cons:
>
> . Transfer is messy.
>
> What happens to all the entry points and properties to the context object
> left in the main page?
> What happens to the "cavnas" parameter on the transferred context?
> How do you synchronize the main page setting canvas.width or canvas.height
> with the worker trying to render to it?
>
>
> *) Create a context directly in a Worker (or anywhere for that matter)
>
> As in "var gl = new WebGLRenderingContext"
>
> Pros:
>
> . ???
>
> Cons:
>
> . requires defining how the backbuffer size is set.
>    Maybe there is no backbuffer for a directly created
> WebGLRenderingContext?
> . if there is no backbuffer then using one of these contexts to draw into a
> texture
>   or 2d context is problematic

This alternative seems like the one that can be moved forward most
easily, since essentially all of the changes would be within the WebGL
spec.

 - Extend the spec to support context sharing. (Each
WebGLRenderingContext points to an opaque WebGLContextGroup object.)
 - Define structured cloning semantics for WebGLContextGroup.
 - Add a constructor or factory method to WebGLRenderingContext
allowing creation with a WebGLContextGroup. Contexts created in this
manner would have no back buffer by default. (FBOs could still be used
to do rendering with the context.)
 - Allow some or all of the WebGLObject types (textures, etc.) to be
either copied during structured cloning or transferred.

Then the worker can at least upload textures to the GPU completely
asynchronously from the main thread, and inform the main thread via
postMessage when they have finished uploading. That seems to be a
tractable first step and one that would already have immediate
benefits for developers.

-Ken


> *) Create an offscreen canvas like object minus HTMLElement parts
>
> Pros:
>
> . same or nearly the API as already exists for canvas
> . flexible. getContext can return different kinds of context. Maybe only
> "webgl" for now?
>
> Cons:
>
> . ???
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Gregg Tavares (wrk)


On Wed, May 16, 2012 at 12:42 PM, Kenneth Russell <[hidden email]> wrote:
On Wed, May 16, 2012 at 12:30 PM, Gregg Tavares (勤) <[hidden email]> wrote:
> So how to take this forward?
>
> My #1 priority is to get WebGL in workers. Lots of developers have expressed
> a need for this from decoding compressed textures in a worker to offloading
> thousands of draw calls per frame to a worker. WebGL already defines sharing
> mechanisms so at least for the WebGL case I don't need to solve how to
> display anything back in the main worker. The worker can draw to a texture.
> The main page can use that texture. WebGL/OpenGL already define that
> relationship.
>
> But, to get WebGL in a worker I need a way to get a WebGLRenderingContext
> into a worker.
>
> Some Ideas
>
> *) Create context in main page, pass to worker
>
> Pros:
>
> . No new APIs or objects.
>
> Cons:
>
> . Transfer is messy.
>
> What happens to all the entry points and properties to the context object
> left in the main page?
> What happens to the "cavnas" parameter on the transferred context?
> How do you synchronize the main page setting canvas.width or canvas.height
> with the worker trying to render to it?
>
>
> *) Create a context directly in a Worker (or anywhere for that matter)
>
> As in "var gl = new WebGLRenderingContext"
>
> Pros:
>
> . ???
>
> Cons:
>
> . requires defining how the backbuffer size is set.
>    Maybe there is no backbuffer for a directly created
> WebGLRenderingContext?
> . if there is no backbuffer then using one of these contexts to draw into a
> texture
>   or 2d context is problematic

This alternative seems like the one that can be moved forward most
easily, since essentially all of the changes would be within the WebGL
spec.

 - Extend the spec to support context sharing. (Each
WebGLRenderingContext points to an opaque WebGLContextGroup object.)
 - Define structured cloning semantics for WebGLContextGroup.
 - Add a constructor or factory method to WebGLRenderingContext
allowing creation with a WebGLContextGroup. Contexts created in this
manner would have no back buffer by default. (FBOs could still be used
to do rendering with the context.)
 - Allow some or all of the WebGLObject types (textures, etc.) to be
either copied during structured cloning or transferred.

Then the worker can at least upload textures to the GPU completely
asynchronously from the main thread, and inform the main thread via
postMessage when they have finished uploading. That seems to be a
tractable first step and one that would already have immediate
benefits for developers.

That problem I have with this path is the cons mentioned above. It's a dead end.

With real contexts you can do this

    webglcontext1.texImage2D(..., webglcontext2.canvas);

and this

    context2d.drawImage(webgl.canvas, ...)

But with a context created with new WebGLRenderingContext you can't do anything like that because size of the backbuffer is not defined. Eventually we'll want to support operation like that and we'll likely end up with something like DrawingSurface or CanvasSurface or OffscreenCanvas. At that point this ability to go "new WebGLRenderingContext" will just be left over cruft.

Also we'll have to define how to addEventListner for WebGLRenderContext for listening for lost context or for async context creation (yea, less important on workers). That stuff is currently not on WebGLRenderingContext. It seems like it should stay off if it.






 

-Ken


> *) Create an offscreen canvas like object minus HTMLElement parts
>
> Pros:
>
> . same or nearly the API as already exists for canvas
> . flexible. getContext can return different kinds of context. Maybe only
> "webgl" for now?
>
> Cons:
>
> . ???
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

妟婹
On Wed, May 16, 2012 at 1:11 PM, Gregg Tavares (勤) <[hidden email]> wrote:

>
>
> On Wed, May 16, 2012 at 12:42 PM, Kenneth Russell <[hidden email]> wrote:
>>
>> On Wed, May 16, 2012 at 12:30 PM, Gregg Tavares (勤) <[hidden email]>
>> wrote:
>> > So how to take this forward?
>> >
>> > My #1 priority is to get WebGL in workers. Lots of developers have
>> > expressed
>> > a need for this from decoding compressed textures in a worker to
>> > offloading
>> > thousands of draw calls per frame to a worker. WebGL already defines
>> > sharing
>> > mechanisms so at least for the WebGL case I don't need to solve how to
>> > display anything back in the main worker. The worker can draw to a
>> > texture.
>> > The main page can use that texture. WebGL/OpenGL already define that
>> > relationship.
>> >
>> > But, to get WebGL in a worker I need a way to get a
>> > WebGLRenderingContext
>> > into a worker.
>> >
>> > Some Ideas
>> >
>> > *) Create context in main page, pass to worker
>> >
>> > Pros:
>> >
>> > . No new APIs or objects.
>> >
>> > Cons:
>> >
>> > . Transfer is messy.
>> >
>> > What happens to all the entry points and properties to the context
>> > object
>> > left in the main page?
>> > What happens to the "cavnas" parameter on the transferred context?
>> > How do you synchronize the main page setting canvas.width or
>> > canvas.height
>> > with the worker trying to render to it?
>> >
>> >
>> > *) Create a context directly in a Worker (or anywhere for that matter)
>> >
>> > As in "var gl = new WebGLRenderingContext"
>> >
>> > Pros:
>> >
>> > . ???
>> >
>> > Cons:
>> >
>> > . requires defining how the backbuffer size is set.
>> >    Maybe there is no backbuffer for a directly created
>> > WebGLRenderingContext?
>> > . if there is no backbuffer then using one of these contexts to draw
>> > into a
>> > texture
>> >   or 2d context is problematic
>>
>> This alternative seems like the one that can be moved forward most
>> easily, since essentially all of the changes would be within the WebGL
>> spec.
>>
>>  - Extend the spec to support context sharing. (Each
>> WebGLRenderingContext points to an opaque WebGLContextGroup object.)
>>  - Define structured cloning semantics for WebGLContextGroup.
>>  - Add a constructor or factory method to WebGLRenderingContext
>> allowing creation with a WebGLContextGroup. Contexts created in this
>> manner would have no back buffer by default. (FBOs could still be used
>> to do rendering with the context.)
>>  - Allow some or all of the WebGLObject types (textures, etc.) to be
>> either copied during structured cloning or transferred.
>>
>> Then the worker can at least upload textures to the GPU completely
>> asynchronously from the main thread, and inform the main thread via
>> postMessage when they have finished uploading. That seems to be a
>> tractable first step and one that would already have immediate
>> benefits for developers.
>
>
> That problem I have with this path is the cons mentioned above. It's a dead
> end.
>
> With real contexts you can do this
>
>     webglcontext1.texImage2D(..., webglcontext2.canvas);
>
> and this
>
>     context2d.drawImage(webgl.canvas, ...)
>
> But with a context created with new WebGLRenderingContext you can't do
> anything like that because size of the backbuffer is not defined. Eventually
> we'll want to support operation like that and we'll likely end up with
> something like DrawingSurface or CanvasSurface or OffscreenCanvas. At that
> point this ability to go "new WebGLRenderingContext" will just be left over
> cruft.
>
> Also we'll have to define how to addEventListner for WebGLRenderContext for
> listening for lost context or for async context creation (yea, less
> important on workers). That stuff is currently not on WebGLRenderingContext.
> It seems like it should stay off if it.

These are all good points, and I agree that the ideal fix would be to
use the Canvas element (or DrawingSurface, etc.) from a worker, rather
than doing a WebGL-specific hack. Given how long discussions have been
ongoing on this topic, though, I think this may just be too large a
single step to take.

Rather than add a constructor to WebGLRenderingContext that would need
to remain there permanently, perhaps a factory method with a vendor
prefix could be added (e.g.
"WebGLRenderingContext.webkitCreateOffscreenContext",
"WebGLRenderingContext.mozCreateOffscreenContext"). The long-term plan
could be to remove the factory method once a better solution is
reached, as opposed to removing the vendor prefixes.

This would at least allow some applications to explore the use of
workers with WebGL, see whether this addresses any existing issues,
and expose new issues.

-Ken



>>
>>
>> -Ken
>>
>>
>> > *) Create an offscreen canvas like object minus HTMLElement parts
>> >
>> > Pros:
>> >
>> > . same or nearly the API as already exists for canvas
>> > . flexible. getContext can return different kinds of context. Maybe only
>> > "webgl" for now?
>> >
>> > Cons:
>> >
>> > . ???
>> >
>> >
>> >
>> >
>
>

Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Glenn Maynard
In reply to this post by Gregg Tavares (wrk)
On Wed, May 16, 2012 at 2:30 PM, Gregg Tavares (勤) <[hidden email]> wrote:
Some Ideas

*) Create context in main page, pass to worker

Transferring an active context from one thread to another (at the implementation level) can be hard.  It's much simpler to only create new the contexts in the thread they'll be used, never allowing contexts to be moved from thread to thread.

Here's a first-pass approach:

1: Add HTMLCanvasElement.getBackbuffer, which returns a lightweight Backbuffer interface associated with the HTMLCanvasElement.  This has only one method, getContext, which is equivalent to canvas.getContext.  (This is exactly like the suggestion for making images available in threads.)
2: Backbuffer can be transferred from one thread to another, using the transfer mechanism.  On transfer, any WebGL contexts (and other context types) in the thread associated with the same backbuffer are destroyed (putting exactly what "destroyed" means aside for later).  Additionally, when a Backbuffer is transferred, the underlying backbuffer of the HTMLCanvasElement is considered owned by the new thread.  All instances of Backbuffer for that HTMLCanvasElement, including newly-created ones, are neutered unless the backbuffer is transferred back to the UI thread.
3: As long as the backbuffer of an HTMLCanvasElement is owned by a thread other than the UI thread, calls to the following methods on the associated HTMLCanvasElement raise an exception: getBackbuffer, getContext, toBlob, toDataURL.  This takes effect synchronously, as soon as transfer happens (right when you postMessage the Backbuffer somewhere else).
4: When a Worker receives a Backbuffer, it can call Backbuffer.getContext to create a WebGLContext (with exactly the same rules as HTMLCanvasElement.getContext).  The results of rendering to that context are visible in the associated HTMLCanvasElement in the main thread.

This ensures that only one thread can ever have an open context for any backbuffer, and that pixel readback functions (toBlob and toDataURL) can't expose the asynchronous nature of what's going on.  The transfer mechanics are rough and need refining.

For offscreen rendering in a thread, just allow constructing Backbuffer; for example,

var backbuffer = new Backbuffer(1024, 768);
var ctx = backbuffer.getContext("webgl");
backbuffer.resize(1920, 1200);

Backbuffer.resize is equivalent to resizing a Canvas, and only available for Backbuffers you create yourself, not for ones created via HTMLCanvasElement.getBackBuffer.

One problem that I havn't attempted to solve here: when the HTMLCanvasElement is resized in the UI thread, it results in changes to drawingBufferWidth/drawingBufferHeight.  This would expose asynchronous behavior.  Instead, it would probably need to apply the change (from the thread's perspective) in a queued task, so you have to return to the event loop for it to be applied.

If a thread is killed by the browser (eg. due to a CPU quota), the backbuffers it owns are orphaned; you can no longer create contexts for it.  You need to create a new Canvas.  This isn't great, but workers don't really seem to try to make it possible to recover from this anyway.

interface CanvasBackbuffer {
    object? getContext(DOMString contextId, any... args);
}

[Constructor(unsigned long width, unsigned long height)]
interface Backbuffer : CanvasBackbuffer {
    void resize(unsigned long width, unsigned long height);
}

interface HTMLCanvasElement : HTMLElement {
    CanvasBackbuffer getBackbuffer();
}

The Backbuffer ctor is available in workers.

Importantly, you can start by just implementing the Backbuffer constructor.  That would only allow the simpler case of offscreen rendering.

--
Glenn Maynard

Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Robert O'Callahan-3
That sounds like a reasonable approach to me. One thing missing is that Workers would need something like requestAnimationFrame.

Any proposal that requires passing messages to the main thread to get something on the screen fails to satisfy the huge need for games to be able to get steady frame rates in the face of main-thread latency.

Rob
--
“You have heard that it was said, ‘Love your neighbor and hate your enemy.’ But I tell you, love your enemies and pray for those who persecute you, that you may be children of your Father in heaven. ... If you love those who love you, what reward will you get? Are not even the tax collectors doing that? And if you greet only your own people, what are you doing more than others?" [Matthew 5:43-47]

Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Ian Hickson
In reply to this post by Gregg Tavares (wrk)
On Mon, 14 May 2012, Gregg Tavares (�~K�) wrote:
>
> I'd like to work on exposing something like CANVAS to web workers.
>
> Ideally how over it works I'd like to be able to
>
> *) get a 2d context in a web worker
> *) get a WebGL context in a web worker
> *) download images in a web worker and the images with both 2d contexts and
> WebGL contexts

I've now specced something like this; for details, see:

   http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Nov/0199.html

--
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'
Reply | Threaded
Open this post in threaded view
|

Re: exposing CANVAS or something like it to Web Workers

Charles Pritchard-2
On Nov 16, 2012, at 1:50 PM, Ian Hickson <[hidden email]> wrote:

> On Mon, 14 May 2012, Gregg Tavares (�~K�) wrote:
>>
>> I'd like to work on exposing something like CANVAS to web workers.
>>
>> Ideally how over it works I'd like to be able to
>>
>> *) get a 2d context in a web worker
>> *) get a WebGL context in a web worker
>> *) download images in a web worker and the images with both 2d contexts and
>> WebGL contexts
>
> I've now specced something like this; for details, see:
>
>   http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Nov/0199.html

Seems like we might use requestAnimationFrame in the main thread to postMessage to the worker as an alternative to using setInterval in workers for repaints.

-Charles
123