Quantcast

Comment on minutes ## With Credentials flag etc

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
25 messages Options
12
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Comment on minutes ## With Credentials flag etc

Tim Berners-Lee
## With Credentials flag
mnot: TimBL's issue, he discussed a bit with annevk; I discussed with annevk as well.
mnot: Tim needs to justify a little more

Sad, I wasn’t in Australia.  But I did describe it in Berlin face-face.

Yves: If you can't figure out whether it's ??? then you can't figure out whether the server is .... then you have to link to not just URI but URI plus credentials.
mnot: ... saying not consistent with Web architecture to have fetch have a flag (with credentials) for whether it's cross-origin; wants it to be calculated automatically
Yves: ... a bit like an API
mnot: you didn't have to do that
mnot: so why is this architecturally bad?  Strength of URI is that stick URI into application.  But application does know context.  Maybe Tim misunderstands nature of fetch; not really and end-user-facing API.

I do not think I misunderstand the nature of fetch().
The nature of fetch is to look up a URI given the URI and nothing else.
Fetch (or XHR)  is called by many many pieces of code, sometimes developer code, sometime library code.
Never the user. 

Fetch just asks for a URI to be fetched.
Everything is handled by the browser code, the developer goes not have to deal with
- authentication — the developer does not have to give the passwords etc
So in general the called of fetch() does NOT know whether authentication will be needed, and does not care. 
So the caller of fetch does not, in the general case, know whether to set any withCredentials flag.


Yves: different from regular web page, running in specific .??? context
plinss: the with credentials flag is about how the url is used

All the information about how the URL is used is in the specs, and in the URL.
That is web architecture 101.

mnot: legitimate to have something about how URL is being used

Nope, not legitimate.  
Every time someone adds something like this, like “force content-type” flags, it means everywhere the URI is stored throughout the system

mnot: but we can't resolve this without Tim in the room

Sad

Yves: .... is it to hide the fact that it's security through obscurity?
alex: ???
Yves: would it be possibe to have 2 different kinds of errors:  unknown not-transport-level error, vs. could reach server
Travis: sounds like layering violation
alex: if I'm trying to provide a UI, I'd like to say what might be going wrong
travis: at that level, you know what you need
alex: I mean Web site ui, not browser ui.
Yves: Is it good to have didnt-reach-server vs. didnt-get-what-wanted exception?
alex: to be clear, we're talking about special case of cross-origin non-CORS requests
travis: Tim would say that with-credentials part not included ...
Yves: If not able to figure out whether real network error, then libraries would ... having proper error miht avoid leaking that information
travis: scenario is I request resource and get denied -- then maybe I need to do a CORS request?
Yves: You don't need to know if it was denied or not found -- just reach server and there was an error.

If you hide from the script what is going on

- You prevent fishing I suppose

but

- you make it really hard for people to get systems working, with the user just hanging typically when th script sees readyState = 4 and  a status of zero which is not in the spec

- -you make it impossible for the script writer to tell the user what is going on, “Your data source needs to upgrade their apache CORS module to version 87"

Yves: You have a core interaction that failed -- at fetch level coul djust report that getting what you wanted failed
mnot: would anything but CORS produce this kind of error?
Yves: you don't need to know if it's a CORS error to hide the fact that it was really denied,   Goal of fetch is to hide that info; everything is a network error.

“Network error” - like a TCP timeout of an ethernet interface turned off?
I imagine that a malware script could tell the difference between these, just hard for the user or a safe app to.
Fetch should be able to return nested errors like

HTTP Error: TCP error opening TCP connection:
 TCP Error: could not start new connection because of;
  Ethernet problem: you have no  ethernet interfaces powered up

Really important to distinguish for example between the DNS knowing the name you asked for is not in it, and the DNS not being reachable.

alex: XHR has same behavior.  Scenario:  trying to cross-origin request, with with-credentials flag.  Even if sets allow-access...: *, still won't get response or know what happened.

Yes - I get these problems with XHR

travis: not what I understood
alex: that's what Tim says in the issue
mnot: I wonder if tim knows it's not an end-user-facing flag

Clearly.
The problem is there is no party is happy to faced by that flag.

alex: it's exposed through XHR and fetch
alex: that's how this came up.
travis: XHR was not a networking-level problem, ... application layer policy that said you're only allowed to talk to your origin
travis: so application layer created other thing that says unless you provide exception and handshake and right headers...
alex: a couple oddities here.  Starts at which requests get CORS and which don't.
alex: some implicitly or explicitly invoke CORS, some don't.  It's defined, but may not be reasonable if you're trying to programmatically access things.
alex: I think issues bottoms out at:  some requests can be made to use CORS, some use by default, some can't be made to use CORS, even if you want them to.  And that is probably the original sin here, but it's fallen off the stack for me which is which.

Are you saying the CORS design is really a kludge of rather arbitrary decisions, rather than a system with well-defined goals and properties?

travis: example of something that needs CORS and will implicitly have CORS enabled is module loading.
Alex: fonts always use CORS
alex: images never use CORS
travis: script is configurable
alex: how?
travis: user can supply the attribute that says to use CORS?  Or maybe not?  Maybe only DOM and not markup.
alex: crossorigin attribute
alex: do images have crossorigin attribute?

So what information is used to set that attribute?
What does the coder of it have to know that the script and the browser code do not know?

travis: link, script, media, import
plinss: crossorigin images and tainted canvases
alex: possible to invoke for images
alex: always for fonts, possible to invoke for script
travis: sounds like we're retrofitting most things that use resources to have the option of using CORS

Maybe we should use a new URI scheme. <- reductio ad absurdum
So this information can be held in bookmarks and caches and histories etc etc

alex: can't tell from issue whether CORS values Access-control-allow-credentials vs. access-control-allow-origin are part of the debate
alex: would love to see network trace and code snippet that's not working for tim
mnot: as I understand it, he just doesn't want to have to send the with-credentials flag when he'se requesting data from a different site
mnot: I think he wants to not have to know.

yes.

alex: I think what's going on here -- if he sets credentials for every request, some will be automatically denied -- don't recall which is which.
alex: maybe we should ask annevk
plinss: get tim and anne on a call at the same time??
alex: would be nice to have an enumeration somelpace of the states, and odd to me that this fetch spec doesn't have anything like that

A flow chart would be peachy, or a machine-readable

mnot: spaghetti

In that case, maybe the whole things needs to be rethought from scratch.

travis: cross-origin for anchors?
mnot: we need a fetch explainer doc
alex: I don't think it exists.
mnot: I've thought about it, but worry it'll change on me.
dbaron: what does crossorigin on script do
travis: lets you get script error messages




signature.asc (858 bytes) Download Attachment
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Mark Nottingham-2

> On 19 Jan 2016, at 8:36 am, Tim Berners-Lee <[hidden email]> wrote:
>
>> ## With Credentials flag
>> https://github.com/w3ctag/spec-reviews/issues/76
>> mnot: TimBL's issue, he discussed a bit with annevk; I discussed with annevk as well.
>> mnot: Tim needs to justify a little more
>
> Sad, I wasn’t in Australia.

We were too.

> But I did describe it in Berlin face-face.

Yep; we were continuing that discussion (albeit with somewhat stale memories).

>> Yves: If you can't figure out whether it's ??? then you can't figure out whether the server is .... then you have to link to not just URI but URI plus credentials.
>> mnot: ... saying not consistent with Web architecture to have fetch have a flag (with credentials) for whether it's cross-origin; wants it to be calculated automatically
>> Yves: ... a bit like an API
>> mnot: you didn't have to do that
>> mnot: so why is this architecturally bad?  Strength of URI is that stick URI into application.  But application does know context.  Maybe Tim misunderstands nature of fetch; not really and end-user-facing API.
>
> I do not think I misunderstand the nature of fetch().
> The nature of fetch is to look up a URI given the URI and nothing else.
> Fetch (or XHR)  is called by many many pieces of code, sometimes developer code, sometime library code.
> Never the user.
>
> Fetch just asks for a URI to be fetched.
> Everything is handled by the browser code, the developer goes not have to deal with
> - authentication — the developer does not have to give the passwords etc
> So in general the called of fetch() does NOT know whether authentication will be needed, and does not care.
> So the caller of fetch does not, in the general case, know whether to set any withCredentials flag.

Yep, we came to the same place later in the conversation; withCredentials is exposed by XHR, which caused us confusion too.


>> Yves: different from regular web page, running in specific .??? context
>> plinss: the with credentials flag is about how the url is used
>
> All the information about how the URL is used is in the specs, and in the URL.
> That is web architecture 101.
>
>> mnot: legitimate to have something about how URL is being used
>
> Nope, not legitimate.  
> Every time someone adds something like this, like “force content-type” flags, it means everywhere the URI is stored throughout the system
>
>> mnot: but we can't resolve this without Tim in the room
>
> Sad
>
>> Yves: .... is it to hide the fact that it's security through obscurity?
>> alex: ???
>> Yves: would it be possibe to have 2 different kinds of errors:  unknown not-transport-level error, vs. could reach server
>> Travis: sounds like layering violation
>> alex: if I'm trying to provide a UI, I'd like to say what might be going wrong
>> travis: at that level, you know what you need
>> alex: I mean Web site ui, not browser ui.
>> Yves: Is it good to have didnt-reach-server vs. didnt-get-what-wanted exception?
>> alex: to be clear, we're talking about special case of cross-origin non-CORS requests
>> travis: Tim would say that with-credentials part not included ...
>> Yves: If not able to figure out whether real network error, then libraries would ... having proper error miht avoid leaking that information
>> travis: scenario is I request resource and get denied -- then maybe I need to do a CORS request?
>> Yves: You don't need to know if it was denied or not found -- just reach server and there was an error.
>
> If you hide from the script what is going on
>
> - You prevent fishing I suppose
>
> but
>
> - you make it really hard for people to get systems working, with the user just hanging typically when th script sees readyState = 4 and  a status of zero which is not in the spec
>
> - -you make it impossible for the script writer to tell the user what is going on, “Your data source needs to upgrade their apache CORS module to version 87"
>
>> Yves: You have a core interaction that failed -- at fetch level coul djust report that getting what you wanted failed
>> mnot: would anything but CORS produce this kind of error?
>> Yves: you don't need to know if it's a CORS error to hide the fact that it was really denied,   Goal of fetch is to hide that info; everything is a network error.
>
> “Network error” - like a TCP timeout of an ethernet interface turned off?
> I imagine that a malware script could tell the difference between these, just hard for the user or a safe app to.
> Fetch should be able to return nested errors like
>
> HTTP Error: TCP error opening TCP connection:
>  TCP Error: could not start new connection because of;
>   Ethernet problem: you have no  ethernet interfaces powered up
>
> Really important to distinguish for example between the DNS knowing the name you asked for is not in it, and the DNS not being reachable.
>
>> alex: XHR has same behavior.  Scenario:  trying to cross-origin request, with with-credentials flag.  Even if sets allow-access...: *, still won't get response or know what happened.
>
> Yes - I get these problems with XHR
>
>> travis: not what I understood
>> alex: that's what Tim says in the issue
>> mnot: I wonder if tim knows it's not an end-user-facing flag
>
> Clearly.
> The problem is there is no party is happy to faced by that flag.
>
>> alex: it's exposed through XHR and fetch
>> alex: that's how this came up.
>> travis: XHR was not a networking-level problem, ... application layer policy that said you're only allowed to talk to your origin
>> travis: so application layer created other thing that says unless you provide exception and handshake and right headers...
>> alex: a couple oddities here.  Starts at which requests get CORS and which don't.
>> alex: some implicitly or explicitly invoke CORS, some don't.  It's defined, but may not be reasonable if you're trying to programmatically access things.
>> alex: I think issues bottoms out at:  some requests can be made to use CORS, some use by default, some can't be made to use CORS, even if you want them to.  And that is probably the original sin here, but it's fallen off the stack for me which is which.
>
> Are you saying the CORS design is really a kludge of rather arbitrary decisions, rather than a system with well-defined goals and properties?
>
>> travis: example of something that needs CORS and will implicitly have CORS enabled is module loading.
>> Alex: fonts always use CORS
>> alex: images never use CORS
>> travis: script is configurable
>> alex: how?
>> travis: user can supply the attribute that says to use CORS?  Or maybe not?  Maybe only DOM and not markup.
>> alex: crossorigin attribute
>> alex: do images have crossorigin attribute?
>
> So what information is used to set that attribute?
> What does the coder of it have to know that the script and the browser code do not know?
>
>> travis: link, script, media, import
>> plinss: crossorigin images and tainted canvases
>> alex: possible to invoke for images
>> alex: always for fonts, possible to invoke for script
>> travis: sounds like we're retrofitting most things that use resources to have the option of using CORS
>
> Maybe we should use a new URI scheme. <- reductio ad absurdum
> So this information can be held in bookmarks and caches and histories etc etc
>
>> alex: can't tell from issue whether CORS values Access-control-allow-credentials vs. access-control-allow-origin are part of the debate
>> alex: would love to see network trace and code snippet that's not working for tim
>> mnot: as I understand it, he just doesn't want to have to send the with-credentials flag when he'se requesting data from a different site
>> mnot: I think he wants to not have to know.
>
> yes.
>
>> alex: I think what's going on here -- if he sets credentials for every request, some will be automatically denied -- don't recall which is which.
>> alex: maybe we should ask annevk
>> plinss: get tim and anne on a call at the same time??
>> alex: would be nice to have an enumeration somelpace of the states, and odd to me that this fetch spec doesn't have anything like that
>
> A flow chart would be peachy, or a machine-readable
>
>> mnot: spaghetti
>
> In that case, maybe the whole things needs to be rethought from scratch.

... or at least the motivations behind the decisions explained. It's pretty impenetrable now, and even security folks don't profess to know all of the details behind CORS any more.


>> travis: cross-origin for anchors?
>> mnot: we need a fetch explainer doc
>> alex: I don't think it exists.
>> mnot: I've thought about it, but worry it'll change on me.
>> dbaron: what does crossorigin on script do
>> travis: lets you get script error messages


--
Mark Nottingham   https://www.mnot.net/


Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Tim Berners-Lee
Here is an example of this issue in the field as captured from a genuine live overheard conversation on a gitter channel. 



csarven
Jan 14 22:00
@deiu / @nicola / all.. Could someone please try to help me figure this out: Go to https://dokie.li/ and 'Sign in' (from the menu at the top right corner) using your WebID in both Firefox and Chrome/ium. Keep the dev console window open and see if you get a 401 on your preferences URL in either one of those browsers. If your prefs is at databox.me that might help understand better. I am able to authenticate (using http and https WebIDs), however when it (simplerdf) tries to get the prefs, Firefox gives me a 401, and Chrome is a 200. The header requests appear to be exactly the same in both browsers.
csarven
Jan 14 22:14
In Firefox, I am able GET the prefs in a different tab (for both of the WebID's prefs which are at databox). So I'm suspecting that the issue is closer to SimpleRDF's request in Firefox somehow.
deiu
Jan 14 22:15
Are you using withCredentials = true flag in your ajax request?
csarven
Jan 14 22:16
I was just going to say that.. SimpleRDF doesn't.
deiu
Jan 14 22:16
It should
csarven
Jan 14 22:16
Right.. I thought so: rdf-ext/rdf-ext#52 :smile:
timbl
09:28
...
timbl
09:35
This is exactly the issue the TAG were mulling over in my absence in Melbourne. The problem is the library simpleRDF is screwed either way — if it doesn’t set the flag then you don’t get to authenticate, if it does set it then the borwser blocks access to resources with a wildcard CORS header. 

signature.asc (858 bytes) Download Attachment
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Brian Kardell
In reply to this post by Tim Berners-Lee


On Mon, Jan 18, 2016 at 4:36 PM, Tim Berners-Lee <[hidden email]> wrote:
## With Credentials flag
mnot: TimBL's issue, he discussed a bit with annevk; I discussed with annevk as well.
mnot: Tim needs to justify a little more

Sad, I wasn’t in Australia.  But I did describe it in Berlin face-face.

Yves: If you can't figure out whether it's ??? then you can't figure out whether the server is .... then you have to link to not just URI but URI plus credentials.
mnot: ... saying not consistent with Web architecture to have fetch have a flag (with credentials) for whether it's cross-origin; wants it to be calculated automatically
Yves: ... a bit like an API
mnot: you didn't have to do that
mnot: so why is this architecturally bad?  Strength of URI is that stick URI into application.  But application does know context.  Maybe Tim misunderstands nature of fetch; not really and end-user-facing API.

I do not think I misunderstand the nature of fetch().
The nature of fetch is to look up a URI given the URI and nothing else.
Fetch (or XHR)  is called by many many pieces of code, sometimes developer code, sometime library code.
Never the user. 

Fetch just asks for a URI to be fetched.
Everything is handled by the browser code, the developer goes not have to deal with
- authentication — the developer does not have to give the passwords etc
So in general the called of fetch() does NOT know whether authentication will be needed, and does not care. 
So the caller of fetch does not, in the general case, know whether to set any withCredentials flag.


Yves: different from regular web page, running in specific .??? context
plinss: the with credentials flag is about how the url is used

All the information about how the URL is used is in the specs, and in the URL.
That is web architecture 101.

mnot: legitimate to have something about how URL is being used

Nope, not legitimate.  
Every time someone adds something like this, like “force content-type” flags, it means everywhere the URI is stored throughout the system

mnot: but we can't resolve this without Tim in the room

Sad


I'm hesitant to add noise to this conversation but I'm very confused by it and I think, from conversations, that I am not alone so I am going to anyway...

Let's assume for a moment, just for the sake of argument, (I'm not sure if this is what Tim is really saying or not but let's just start there) that CORS was a very bad idea and that there is something fundamentally better which didn't require setting this flag and therefore wouldn't be need to be exposed via script API - it'd just "work" (I actually can't imagine how this would be possible but very possibly I am just not being creative enough, in any case, /me waves hand and we assume it is so for purposes here).

Given this assumption, it seems to me that it would not change the fact that millions of websites today function safely because of and rely on this in the real, shipped Web of today.  It is exposed through XHR today and necessary - I don't think you can take that 'out' without breaking significant bits of the Web (importantly to me personally as it would break dozens and dozens of things we've spent years getting deployed for my employer).  The challenge, as I understand it, that Anne took up was to explain "what's down there" underneath all that stuff, including XHR that does all the network browser magic.  I don't think Tim disagrees that this is a laudable goal but if so, that'd be interesting to hear and take the conversation in a very different direction I think.

Thus - if you have to write something at the very low level that has to respect XHR's flag, needs to provide explanation for all of the rest of it (like the fact that some tags request things always with CORS or without) and should (IMO at least) provide enough power that if we wanted to add a new element like image, video or audio tomorrow it would be possible to make that work as a custom element and demonstrate its properties... How else could you do all those things?

I guess this is where my confusion lies.  CORS seems, for good or for bad, to be unexplained and currently necessary magic in the platform that can't just be removed.  As Anne said, once it's there it can be hidden it at higher layers - you can provide better additive abstractions for future use (all the tags do this already and it's possible that some higher level imperative API could too if you could come up with some rationale as to how), but it doesn't seem like there's an option to do much else regardless of its quality?

--
Brian Kardell :: @briankardell 
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Jonas Sicking-2
In reply to this post by Mark Nottingham-2
On Mon, Jan 18, 2016 at 3:27 PM, Mark Nottingham <[hidden email]> wrote:
> ... or at least the motivations behind the decisions explained. It's pretty impenetrable now, and even security folks don't profess to know all of the details behind CORS any more.

I'm bummed to hear that aspects of CORS is still confusing even to the
TAG. This stuff likely needs to get documented someplace. I had hoped
that it'd get documented in the spec, but maybe there's a better
place?

I'm not actually sure what exact confusion is being discussed in this
thread is, so I'll address some of the questions I most commonly get.

Q: What does the withCredentials flag do.
A: When it's set to false, requests are sent containing only the
information provided by the requesting website. I.e. the requesting
website's provided URL, headers and request body. The only information
that's added by the browser is information that's hardcoded into the
browser and does not depend on user information. So for example the
user-agent header. No cookies, authentication headers, or client-side
certificates are added by the browser to the request before it is sent
to the target website.

However setting withCredentials to false does not prevent the
requesting website from adding credentials through cookie headers,
authentication headers, URL parameters or any other way that's exposed
through the API which triggered the request.

Additionally, the response data that would normally affect the client
data storage is ignored. So for example set-cookie response headers
are not written to the browsers cookie storage. The returned response
is also not stored in the normal http cache, though if appropriate
browsers may store it in a specific
"CORS-requests-with-withCredentials-set-to-false" cache.

When withCredentials is set to true, requests are handled like
"normal" requests do in a browser. That means that cookies from the
users cookie storage are added based on the target URL. Cached
authentication data is added through the authentication header.

The response is likewise processed like normal, so set-cookie headers
are processed and the response is cached, if appropriate, in the
normal browser http cache.

Q: Why are security checks performed when withCredentials is set to false?
A: Because the user, and the user's browser, might be behind a
firewall and so might be able to access servers which a website would
otherwise not be able to access.

Sadly there is no, to me, known mechanism for detecting if a given
server is behind a firewall.

Q: Is it safe to always set "Access-control-allow-origin: *" on all
responses from a server.
A: As long as the server is connected to the public internet, yes it
is. It does not leak any information that couldn't be loaded using
curl or any other non-browser HTTP client.

If the server is behind a firewall and might contain sensitive
information, the header should not be added.

Q: Why does CORS not allow "Access-control-allow-origin: *" together
with withCredentials=true?
A: It was felt that this was too big of a foot gun.

CORS was designed not long after Adobe had added the crossdomain.xml
feature to Flash Player. The crossdomain.xml feature allows webserver
administrators to easily indicate that the server contains resources
that should be loadable from other origins. The feature only allowed
"normal" requests, i.e. requests similar to ones that CORS makes when
withCredentials=true.

When crossdomain.xml was released many websites opted in allowing data
to be read from other websites in order to share some public data that
was hosted on the server. Unfortunately they forgot that some other
URLs on the server served sensitive user data. The result was that
relatively quickly after the release of the crossdomain.xml multiple
websites leaked sensitive user data.

You could argue that the problem was that crossdomain.xml was
different since it is a per-server configuration file, whereas CORS
uses per-URL headers. Hence CORS would be less prone to server
administrators accidentally opting in to sharing on URLs that server
user sensitive data.

However in practice many (most?) popular web servers allow adding
configuration files which add static http headers to all URLs under a
given directory. So in practice on many servers it would have been
just as easy to make the same mistake with CORS.

Q: Why does CORS not allow listing multiple origins, or allow pattern
matching, in the "Access-control-allow-origin" header?
A: It was felt that if the server uses dynamic server-side logic to
generate responses for a given URL, that they could also then
dynamically generate the appropriate Access-control-allow-origin
header.

For servers that generate static responses you can generally simply
use "Access-control-allow-origin: *". Keep in mind that static
responses can generally be read from non-browser HTTP clients like
curl anyway.

This doesn't account for static responses which are password protected
using either cookies or auth headers. So yeah, our solution here is
not perfect, but we decided to opt for simplicity.

My personal hope was also that generic server modules would be written
to handle CORS support and which would simplify situations like this.
I'm not sure if such modules exist yet or not.

Q: CORS preflights might double the number of needed to talk to
servers that use HTTP APIs that use lots of different URLs. Why not
allow preflight answers to apply to multiple URLs
A: We originally had a solution to this problem. The solution allowed
a preflight to apply to all URLs under a given directory. However it
turned out that some popular contemporary servers handled URLs in
weird ways which made it possible to map any URL into a URL under any
other directory. I.e. a request to A/script.cgi was equivalent to
B/<stuff here>/A/script.cgi. This made it very easy for developers to
misconfigure servers in ways that attackers could take advantage of.

See also the concern above about it being easy for server
administrators to write policies for one URL forgetting that there are
other URLs in the same space which have different requirements.

However there has been some discussions about addressing this problem
on a server-wide basis for requests that have withCredentials=false.
If we do it on a server-wide basis that removes the problems around
strange URL parsing with some servers. And by only doing it for
requests with withCredentials=false it lessens the risk of leaking
sensitive user data.

I'm not sure what the latest status of these discussions are.


If I'm not addressing the concern/questions from the TAG then please
let me know.

I'd really love it if this type of information could make it into the
spec in a way that is understandable to more people.

/ Jonas

Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Daniel Appelquist-4
Just wanted to say thanks, Jonas, for taking the time to write this up. It feels like material that should be posted somewhere more officially – any plans to do so?

Dan


On Thu, 21 Jan 2016 at 01:27 Jonas Sicking <[hidden email]> wrote:
On Mon, Jan 18, 2016 at 3:27 PM, Mark Nottingham <[hidden email]> wrote:
> ... or at least the motivations behind the decisions explained. It's pretty impenetrable now, and even security folks don't profess to know all of the details behind CORS any more.

I'm bummed to hear that aspects of CORS is still confusing even to the
TAG. This stuff likely needs to get documented someplace. I had hoped
that it'd get documented in the spec, but maybe there's a better
place?

I'm not actually sure what exact confusion is being discussed in this
thread is, so I'll address some of the questions I most commonly get.

Q: What does the withCredentials flag do.
A: When it's set to false, requests are sent containing only the
information provided by the requesting website. I.e. the requesting
website's provided URL, headers and request body. The only information
that's added by the browser is information that's hardcoded into the
browser and does not depend on user information. So for example the
user-agent header. No cookies, authentication headers, or client-side
certificates are added by the browser to the request before it is sent
to the target website.

However setting withCredentials to false does not prevent the
requesting website from adding credentials through cookie headers,
authentication headers, URL parameters or any other way that's exposed
through the API which triggered the request.

Additionally, the response data that would normally affect the client
data storage is ignored. So for example set-cookie response headers
are not written to the browsers cookie storage. The returned response
is also not stored in the normal http cache, though if appropriate
browsers may store it in a specific
"CORS-requests-with-withCredentials-set-to-false" cache.

When withCredentials is set to true, requests are handled like
"normal" requests do in a browser. That means that cookies from the
users cookie storage are added based on the target URL. Cached
authentication data is added through the authentication header.

The response is likewise processed like normal, so set-cookie headers
are processed and the response is cached, if appropriate, in the
normal browser http cache.

Q: Why are security checks performed when withCredentials is set to false?
A: Because the user, and the user's browser, might be behind a
firewall and so might be able to access servers which a website would
otherwise not be able to access.

Sadly there is no, to me, known mechanism for detecting if a given
server is behind a firewall.

Q: Is it safe to always set "Access-control-allow-origin: *" on all
responses from a server.
A: As long as the server is connected to the public internet, yes it
is. It does not leak any information that couldn't be loaded using
curl or any other non-browser HTTP client.

If the server is behind a firewall and might contain sensitive
information, the header should not be added.

Q: Why does CORS not allow "Access-control-allow-origin: *" together
with withCredentials=true?
A: It was felt that this was too big of a foot gun.

CORS was designed not long after Adobe had added the crossdomain.xml
feature to Flash Player. The crossdomain.xml feature allows webserver
administrators to easily indicate that the server contains resources
that should be loadable from other origins. The feature only allowed
"normal" requests, i.e. requests similar to ones that CORS makes when
withCredentials=true.

When crossdomain.xml was released many websites opted in allowing data
to be read from other websites in order to share some public data that
was hosted on the server. Unfortunately they forgot that some other
URLs on the server served sensitive user data. The result was that
relatively quickly after the release of the crossdomain.xml multiple
websites leaked sensitive user data.

You could argue that the problem was that crossdomain.xml was
different since it is a per-server configuration file, whereas CORS
uses per-URL headers. Hence CORS would be less prone to server
administrators accidentally opting in to sharing on URLs that server
user sensitive data.

However in practice many (most?) popular web servers allow adding
configuration files which add static http headers to all URLs under a
given directory. So in practice on many servers it would have been
just as easy to make the same mistake with CORS.

Q: Why does CORS not allow listing multiple origins, or allow pattern
matching, in the "Access-control-allow-origin" header?
A: It was felt that if the server uses dynamic server-side logic to
generate responses for a given URL, that they could also then
dynamically generate the appropriate Access-control-allow-origin
header.

For servers that generate static responses you can generally simply
use "Access-control-allow-origin: *". Keep in mind that static
responses can generally be read from non-browser HTTP clients like
curl anyway.

This doesn't account for static responses which are password protected
using either cookies or auth headers. So yeah, our solution here is
not perfect, but we decided to opt for simplicity.

My personal hope was also that generic server modules would be written
to handle CORS support and which would simplify situations like this.
I'm not sure if such modules exist yet or not.

Q: CORS preflights might double the number of needed to talk to
servers that use HTTP APIs that use lots of different URLs. Why not
allow preflight answers to apply to multiple URLs
A: We originally had a solution to this problem. The solution allowed
a preflight to apply to all URLs under a given directory. However it
turned out that some popular contemporary servers handled URLs in
weird ways which made it possible to map any URL into a URL under any
other directory. I.e. a request to A/script.cgi was equivalent to
B/<stuff here>/A/script.cgi. This made it very easy for developers to
misconfigure servers in ways that attackers could take advantage of.

See also the concern above about it being easy for server
administrators to write policies for one URL forgetting that there are
other URLs in the same space which have different requirements.

However there has been some discussions about addressing this problem
on a server-wide basis for requests that have withCredentials=false.
If we do it on a server-wide basis that removes the problems around
strange URL parsing with some servers. And by only doing it for
requests with withCredentials=false it lessens the risk of leaking
sensitive user data.

I'm not sure what the latest status of these discussions are.


If I'm not addressing the concern/questions from the TAG then please
let me know.

I'd really love it if this type of information could make it into the
spec in a way that is understandable to more people.

/ Jonas


signature.asc (465 bytes) Download Attachment
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Jonas Sicking-2
I don't have any good place where I could put it. I'll talk with Anne about working it into the fetch spec.

/ Jonas

On Tue, Jan 26, 2016 at 1:19 AM, Daniel Appelquist <[hidden email]> wrote:
Just wanted to say thanks, Jonas, for taking the time to write this up. It feels like material that should be posted somewhere more officially – any plans to do so?

Dan


On Thu, 21 Jan 2016 at 01:27 Jonas Sicking <[hidden email]> wrote:
On Mon, Jan 18, 2016 at 3:27 PM, Mark Nottingham <[hidden email]> wrote:
> ... or at least the motivations behind the decisions explained. It's pretty impenetrable now, and even security folks don't profess to know all of the details behind CORS any more.

I'm bummed to hear that aspects of CORS is still confusing even to the
TAG. This stuff likely needs to get documented someplace. I had hoped
that it'd get documented in the spec, but maybe there's a better
place?

I'm not actually sure what exact confusion is being discussed in this
thread is, so I'll address some of the questions I most commonly get.

Q: What does the withCredentials flag do.
A: When it's set to false, requests are sent containing only the
information provided by the requesting website. I.e. the requesting
website's provided URL, headers and request body. The only information
that's added by the browser is information that's hardcoded into the
browser and does not depend on user information. So for example the
user-agent header. No cookies, authentication headers, or client-side
certificates are added by the browser to the request before it is sent
to the target website.

However setting withCredentials to false does not prevent the
requesting website from adding credentials through cookie headers,
authentication headers, URL parameters or any other way that's exposed
through the API which triggered the request.

Additionally, the response data that would normally affect the client
data storage is ignored. So for example set-cookie response headers
are not written to the browsers cookie storage. The returned response
is also not stored in the normal http cache, though if appropriate
browsers may store it in a specific
"CORS-requests-with-withCredentials-set-to-false" cache.

When withCredentials is set to true, requests are handled like
"normal" requests do in a browser. That means that cookies from the
users cookie storage are added based on the target URL. Cached
authentication data is added through the authentication header.

The response is likewise processed like normal, so set-cookie headers
are processed and the response is cached, if appropriate, in the
normal browser http cache.

Q: Why are security checks performed when withCredentials is set to false?
A: Because the user, and the user's browser, might be behind a
firewall and so might be able to access servers which a website would
otherwise not be able to access.

Sadly there is no, to me, known mechanism for detecting if a given
server is behind a firewall.

Q: Is it safe to always set "Access-control-allow-origin: *" on all
responses from a server.
A: As long as the server is connected to the public internet, yes it
is. It does not leak any information that couldn't be loaded using
curl or any other non-browser HTTP client.

If the server is behind a firewall and might contain sensitive
information, the header should not be added.

Q: Why does CORS not allow "Access-control-allow-origin: *" together
with withCredentials=true?
A: It was felt that this was too big of a foot gun.

CORS was designed not long after Adobe had added the crossdomain.xml
feature to Flash Player. The crossdomain.xml feature allows webserver
administrators to easily indicate that the server contains resources
that should be loadable from other origins. The feature only allowed
"normal" requests, i.e. requests similar to ones that CORS makes when
withCredentials=true.

When crossdomain.xml was released many websites opted in allowing data
to be read from other websites in order to share some public data that
was hosted on the server. Unfortunately they forgot that some other
URLs on the server served sensitive user data. The result was that
relatively quickly after the release of the crossdomain.xml multiple
websites leaked sensitive user data.

You could argue that the problem was that crossdomain.xml was
different since it is a per-server configuration file, whereas CORS
uses per-URL headers. Hence CORS would be less prone to server
administrators accidentally opting in to sharing on URLs that server
user sensitive data.

However in practice many (most?) popular web servers allow adding
configuration files which add static http headers to all URLs under a
given directory. So in practice on many servers it would have been
just as easy to make the same mistake with CORS.

Q: Why does CORS not allow listing multiple origins, or allow pattern
matching, in the "Access-control-allow-origin" header?
A: It was felt that if the server uses dynamic server-side logic to
generate responses for a given URL, that they could also then
dynamically generate the appropriate Access-control-allow-origin
header.

For servers that generate static responses you can generally simply
use "Access-control-allow-origin: *". Keep in mind that static
responses can generally be read from non-browser HTTP clients like
curl anyway.

This doesn't account for static responses which are password protected
using either cookies or auth headers. So yeah, our solution here is
not perfect, but we decided to opt for simplicity.

My personal hope was also that generic server modules would be written
to handle CORS support and which would simplify situations like this.
I'm not sure if such modules exist yet or not.

Q: CORS preflights might double the number of needed to talk to
servers that use HTTP APIs that use lots of different URLs. Why not
allow preflight answers to apply to multiple URLs
A: We originally had a solution to this problem. The solution allowed
a preflight to apply to all URLs under a given directory. However it
turned out that some popular contemporary servers handled URLs in
weird ways which made it possible to map any URL into a URL under any
other directory. I.e. a request to A/script.cgi was equivalent to
B/<stuff here>/A/script.cgi. This made it very easy for developers to
misconfigure servers in ways that attackers could take advantage of.

See also the concern above about it being easy for server
administrators to write policies for one URL forgetting that there are
other URLs in the same space which have different requirements.

However there has been some discussions about addressing this problem
on a server-wide basis for requests that have withCredentials=false.
If we do it on a server-wide basis that removes the problems around
strange URL parsing with some servers. And by only doing it for
requests with withCredentials=false it lessens the risk of leaking
sensitive user data.

I'm not sure what the latest status of these discussions are.


If I'm not addressing the concern/questions from the TAG then please
let me know.

I'd really love it if this type of information could make it into the
spec in a way that is understandable to more people.

/ Jonas


Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Shane McCarron
In reply to this post by Daniel Appelquist-4
Hear! Hear!  Jonas, that was amazing.  I *nearly* understood a piece of CORS.  I may need to go have a stiff drink.  Thanks!

On Tue, Jan 26, 2016 at 3:19 AM, Daniel Appelquist <[hidden email]> wrote:
Just wanted to say thanks, Jonas, for taking the time to write this up. It feels like material that should be posted somewhere more officially – any plans to do so?

Dan


On Thu, 21 Jan 2016 at 01:27 Jonas Sicking <[hidden email]> wrote:
On Mon, Jan 18, 2016 at 3:27 PM, Mark Nottingham <[hidden email]> wrote:
> ... or at least the motivations behind the decisions explained. It's pretty impenetrable now, and even security folks don't profess to know all of the details behind CORS any more.

I'm bummed to hear that aspects of CORS is still confusing even to the
TAG. This stuff likely needs to get documented someplace. I had hoped
that it'd get documented in the spec, but maybe there's a better
place?

I'm not actually sure what exact confusion is being discussed in this
thread is, so I'll address some of the questions I most commonly get.

Q: What does the withCredentials flag do.
A: When it's set to false, requests are sent containing only the
information provided by the requesting website. I.e. the requesting
website's provided URL, headers and request body. The only information
that's added by the browser is information that's hardcoded into the
browser and does not depend on user information. So for example the
user-agent header. No cookies, authentication headers, or client-side
certificates are added by the browser to the request before it is sent
to the target website.

However setting withCredentials to false does not prevent the
requesting website from adding credentials through cookie headers,
authentication headers, URL parameters or any other way that's exposed
through the API which triggered the request.

Additionally, the response data that would normally affect the client
data storage is ignored. So for example set-cookie response headers
are not written to the browsers cookie storage. The returned response
is also not stored in the normal http cache, though if appropriate
browsers may store it in a specific
"CORS-requests-with-withCredentials-set-to-false" cache.

When withCredentials is set to true, requests are handled like
"normal" requests do in a browser. That means that cookies from the
users cookie storage are added based on the target URL. Cached
authentication data is added through the authentication header.

The response is likewise processed like normal, so set-cookie headers
are processed and the response is cached, if appropriate, in the
normal browser http cache.

Q: Why are security checks performed when withCredentials is set to false?
A: Because the user, and the user's browser, might be behind a
firewall and so might be able to access servers which a website would
otherwise not be able to access.

Sadly there is no, to me, known mechanism for detecting if a given
server is behind a firewall.

Q: Is it safe to always set "Access-control-allow-origin: *" on all
responses from a server.
A: As long as the server is connected to the public internet, yes it
is. It does not leak any information that couldn't be loaded using
curl or any other non-browser HTTP client.

If the server is behind a firewall and might contain sensitive
information, the header should not be added.

Q: Why does CORS not allow "Access-control-allow-origin: *" together
with withCredentials=true?
A: It was felt that this was too big of a foot gun.

CORS was designed not long after Adobe had added the crossdomain.xml
feature to Flash Player. The crossdomain.xml feature allows webserver
administrators to easily indicate that the server contains resources
that should be loadable from other origins. The feature only allowed
"normal" requests, i.e. requests similar to ones that CORS makes when
withCredentials=true.

When crossdomain.xml was released many websites opted in allowing data
to be read from other websites in order to share some public data that
was hosted on the server. Unfortunately they forgot that some other
URLs on the server served sensitive user data. The result was that
relatively quickly after the release of the crossdomain.xml multiple
websites leaked sensitive user data.

You could argue that the problem was that crossdomain.xml was
different since it is a per-server configuration file, whereas CORS
uses per-URL headers. Hence CORS would be less prone to server
administrators accidentally opting in to sharing on URLs that server
user sensitive data.

However in practice many (most?) popular web servers allow adding
configuration files which add static http headers to all URLs under a
given directory. So in practice on many servers it would have been
just as easy to make the same mistake with CORS.

Q: Why does CORS not allow listing multiple origins, or allow pattern
matching, in the "Access-control-allow-origin" header?
A: It was felt that if the server uses dynamic server-side logic to
generate responses for a given URL, that they could also then
dynamically generate the appropriate Access-control-allow-origin
header.

For servers that generate static responses you can generally simply
use "Access-control-allow-origin: *". Keep in mind that static
responses can generally be read from non-browser HTTP clients like
curl anyway.

This doesn't account for static responses which are password protected
using either cookies or auth headers. So yeah, our solution here is
not perfect, but we decided to opt for simplicity.

My personal hope was also that generic server modules would be written
to handle CORS support and which would simplify situations like this.
I'm not sure if such modules exist yet or not.

Q: CORS preflights might double the number of needed to talk to
servers that use HTTP APIs that use lots of different URLs. Why not
allow preflight answers to apply to multiple URLs
A: We originally had a solution to this problem. The solution allowed
a preflight to apply to all URLs under a given directory. However it
turned out that some popular contemporary servers handled URLs in
weird ways which made it possible to map any URL into a URL under any
other directory. I.e. a request to A/script.cgi was equivalent to
B/<stuff here>/A/script.cgi. This made it very easy for developers to
misconfigure servers in ways that attackers could take advantage of.

See also the concern above about it being easy for server
administrators to write policies for one URL forgetting that there are
other URLs in the same space which have different requirements.

However there has been some discussions about addressing this problem
on a server-wide basis for requests that have withCredentials=false.
If we do it on a server-wide basis that removes the problems around
strange URL parsing with some servers. And by only doing it for
requests with withCredentials=false it lessens the risk of leaking
sensitive user data.

I'm not sure what the latest status of these discussions are.


If I'm not addressing the concern/questions from the TAG then please
let me know.

I'd really love it if this type of information could make it into the
spec in a way that is understandable to more people.

/ Jonas




--
Shane McCarron
Managing Director, Applied Testing and Technology, Inc.
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Yves Lafon
In reply to this post by Jonas Sicking-2

> On 21 Jan 2016, at 02:24, Jonas Sicking <[hidden email]> wrote:
>
> On Mon, Jan 18, 2016 at 3:27 PM, Mark Nottingham <[hidden email]> wrote:
>> ... or at least the motivations behind the decisions explained. It's pretty impenetrable now, and even security folks don't profess to know all of the details behind CORS any more.
>
> I'm bummed to hear that aspects of CORS is still confusing even to the
> TAG. This stuff likely needs to get documented someplace. I had hoped
> that it'd get documented in the spec, but maybe there's a better
> place?

[..]

> Q: Why does CORS not allow "Access-control-allow-origin: *" together
> with withCredentials=true?
> A: It was felt that this was too big of a foot gun.
>
> CORS was designed not long after Adobe had added the crossdomain.xml
> feature to Flash Player. The crossdomain.xml feature allows webserver
> administrators to easily indicate that the server contains resources
> that should be loadable from other origins. The feature only allowed
> "normal" requests, i.e. requests similar to ones that CORS makes when
> withCredentials=true.
>
> When crossdomain.xml was released many websites opted in allowing data
> to be read from other websites in order to share some public data that
> was hosted on the server. Unfortunately they forgot that some other
> URLs on the server served sensitive user data. The result was that
> relatively quickly after the release of the crossdomain.xml multiple
> websites leaked sensitive user data.
>
> You could argue that the problem was that crossdomain.xml was
> different since it is a per-server configuration file, whereas CORS
> uses per-URL headers. Hence CORS would be less prone to server
> administrators accidentally opting in to sharing on URLs that server
> user sensitive data.
>
> However in practice many (most?) popular web servers allow adding
> configuration files which add static http headers to all URLs under a
> given directory. So in practice on many servers it would have been
> just as easy to make the same mistake with CORS.

One can find information like https://annevankesteren.nl/2012/12/cors-101
stating that ‘*’ is always good… it is not because of what you just said, that is also why
people can get confused. Thank you for clearing this (during the TAG discussion, the
origin of this decision was not clear).

So to make it always work, you need to echo back the origin in Access-control-allow-origin,
it is very easy to do, and does not require a specific module for Apache.
The issue is that it can lead to careless administrators using that technique
“to make CORS work” and lead to the same issue that led to disallowing “*” with withCredentials.

--
Baroula que barouleras, au tiéu toujou t'entourneras.

        ~~Yves







Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Jonas Sicking-2
On Mon, Feb 1, 2016 at 2:55 AM, Yves Lafon <[hidden email]> wrote:
> One can find information like https://annevankesteren.nl/2012/12/cors-101
> stating that ‘*’ is always good… it is not because of what you just said, that is also why
> people can get confused.

I think we're both trying to say that it's safe to set for any URL
accessible to the public internet. And that the reason that it's safe
is because it only is honored for re requests that were made without
cookies.

> So to make it always work, you need to echo back the origin in Access-control-allow-origin,
> it is very easy to do, and does not require a specific module for Apache.
> The issue is that it can lead to careless administrators using that technique
> “to make CORS work” and lead to the same issue that led to disallowing “*” with withCredentials.

Indeed, it is quite easy to indiscriminately echo back the "Origin"
request header in a "Access-control-allow-origin" response header.
Thereby leaking private data.

I don't know what we could have done on the spec or client side to
lessen that risk though.

If it is still possible to make such improvements without "breaking
the web" I'm all ears.

If it could only have been done when we originally designed the spec
I'm bummed that we missed that opportunity. In theory we could design
a new cross-origin protocol which includes such improvements and let
callers opt in to it.

However in practice I've never heard of people accidentally exposing
undesired data due to the echoing you mention, so I doubt that it'd be
worth designing a new protocol.

/ Jonas

Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Mark Nottingham-2
Hi Jonas,

Thanks for all of the context, it's most helpful.

Could you explain a bit more about how withCredentials fits in?

AIUI, it works with Access-Control-Allow-Credentials (ignoring the interaction with Access-Control-Allow-Origin for now);

1) If both are false, the check succeeds and credentials aren't used.
2) If both are true, the check succeeds and credentials are used.
3) If withCredentials is false and ACAC is true, the check succeeds and credentials aren't used.
4) If withCredentials is true and ACAC is false, the check fails.

... as per <https://www.w3.org/TR/cors/#resource-sharing-check>.

If that's a correct interpretation, the interesting one here is #3. What's not clear to me is whether this is an optimisation or if it's intended to prevent some sort of attack or information leakage.

If it's an optimisation (to avoid the round trip to discover ACAC on what was a simple request), couldn't the browser take the presence of ACAC on the response as a signal that it should have included credentials, and reissue the request? Of course it's not optimal, but it would remove the *need* to set withCredentials to get interoperability.

If it's a security measure, could you (or someone) explain what the thread model is? The CORS spec talks about the possibility of the browser becoming a confused deputy, but it isn't clear (at least to me) how withCredentials mitigates that.

Or, is there some other motivation for this design that I'm missing?

Thanks,





> On 2 Feb 2016, at 8:02 am, Jonas Sicking <[hidden email]> wrote:
>
> On Mon, Feb 1, 2016 at 2:55 AM, Yves Lafon <[hidden email]> wrote:
>> One can find information like https://annevankesteren.nl/2012/12/cors-101
>> stating that ‘*’ is always good… it is not because of what you just said, that is also why
>> people can get confused.
>
> I think we're both trying to say that it's safe to set for any URL
> accessible to the public internet. And that the reason that it's safe
> is because it only is honored for re requests that were made without
> cookies.
>
>> So to make it always work, you need to echo back the origin in Access-control-allow-origin,
>> it is very easy to do, and does not require a specific module for Apache.
>> The issue is that it can lead to careless administrators using that technique
>> “to make CORS work” and lead to the same issue that led to disallowing “*” with withCredentials.
>
> Indeed, it is quite easy to indiscriminately echo back the "Origin"
> request header in a "Access-control-allow-origin" response header.
> Thereby leaking private data.
>
> I don't know what we could have done on the spec or client side to
> lessen that risk though.
>
> If it is still possible to make such improvements without "breaking
> the web" I'm all ears.
>
> If it could only have been done when we originally designed the spec
> I'm bummed that we missed that opportunity. In theory we could design
> a new cross-origin protocol which includes such improvements and let
> callers opt in to it.
>
> However in practice I've never heard of people accidentally exposing
> undesired data due to the echoing you mention, so I doubt that it'd be
> worth designing a new protocol.
>
> / Jonas

--
Mark Nottingham   https://www.mnot.net/


Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Jonas Sicking-2
On Tue, Feb 2, 2016 at 6:44 PM, Mark Nottingham <[hidden email]> wrote:

> Hi Jonas,
>
> Thanks for all of the context, it's most helpful.
>
> Could you explain a bit more about how withCredentials fits in?
>
> AIUI, it works with Access-Control-Allow-Credentials (ignoring the interaction with Access-Control-Allow-Origin for now);
>
> 1) If both are false, the check succeeds and credentials aren't used.
> 2) If both are true, the check succeeds and credentials are used.
> 3) If withCredentials is false and ACAC is true, the check succeeds and credentials aren't used.
> 4) If withCredentials is true and ACAC is false, the check fails.
>
> ... as per <https://www.w3.org/TR/cors/#resource-sharing-check>.
>
> If that's a correct interpretation, the interesting one here is #3. What's not clear to me is whether this is an optimisation or if it's intended to prevent some sort of attack or information leakage.
>
> If it's an optimisation (to avoid the round trip to discover ACAC on what was a simple request), couldn't the browser take the presence of ACAC on the response as a signal that it should have included credentials, and reissue the request? Of course it's not optimal, but it would remove the *need* to set withCredentials to get interoperability.

It is not to mitigate a threat, at least not primarily.

It is to avoid having to issue two requests. I think calling it an
"optimization" is understating it a bit.

Not only would issuing two requests many times likely be so slow that
people would likely opt to use other solutions, such as JSONP.

The other problem is that for requests that have side effects, issuing
the request twice could of course be problematic. This applies to GET
requests as well unfortunately since unfortunately they in practice
often do have side effects.

It's not clear to me how big this problem would be in practice though.
For request that has side effects and that uses cookies, it is likely
common that those cookies are used for authentication and thus issuing
the request without cookies would result in a denied request and no
side effects (maybe other than logging an error).

So in that case issuing two requests, where the first one is without
credentials, would be fine.

But if cookies are used for other things than authentication, then
issuing the request twice seems like it could easily be a problem.

For that reason, I think re-issuing the request when an "ACAC: true"
is returned would be bad (not to mention likely not web compatible at
this point).

All of this was debated while we did the initial CORS design.

What I don't remember debating, and which might work, would be to add
a different header which explicitly has the meaning "if this request
was issued without credentials, please re-issue this request with
credentials".

Then leave it up to the server administrator to make sure that the
initial, credential-less, request doesn't create side effects.


I could see some privacy concerns with this though. Say that website A
wants to use some REST API which website B provides. Currently website
A can do this without exposing to B that the user is using website A.
Simply issue CORS requests with .withCredentials is set to false.

If we add a "please re-issue with credentials" header, then that means
that website B can now figure out the users identity, assuming the
user is logged in with B, by asking the request to be re-issued.

Again, I don't know how big of a problem this would be.


I'm also not sure how popular this feature would be with developers. I
suspect most of them would prefer to use libraries which enable them
to set .withCredentials to true, rather than pay the cost of issuing
two requests.

/ Jonas

Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Mark Nottingham-2
Thanks, Jonas - that's (again) very helpful.

One thing this brings to mind is the current discussion on an origin-level preflight:
  https://github.com/whatwg/fetch/issues/210

If such a thing were defined, it'd provide a more reasonable way for sites to opt into credentials (if the mechanism allows that).

Cheers,


> On 4 Feb 2016, at 5:17 am, Jonas Sicking <[hidden email]> wrote:
>
> On Tue, Feb 2, 2016 at 6:44 PM, Mark Nottingham <[hidden email]> wrote:
>> Hi Jonas,
>>
>> Thanks for all of the context, it's most helpful.
>>
>> Could you explain a bit more about how withCredentials fits in?
>>
>> AIUI, it works with Access-Control-Allow-Credentials (ignoring the interaction with Access-Control-Allow-Origin for now);
>>
>> 1) If both are false, the check succeeds and credentials aren't used.
>> 2) If both are true, the check succeeds and credentials are used.
>> 3) If withCredentials is false and ACAC is true, the check succeeds and credentials aren't used.
>> 4) If withCredentials is true and ACAC is false, the check fails.
>>
>> ... as per <https://www.w3.org/TR/cors/#resource-sharing-check>.
>>
>> If that's a correct interpretation, the interesting one here is #3. What's not clear to me is whether this is an optimisation or if it's intended to prevent some sort of attack or information leakage.
>>
>> If it's an optimisation (to avoid the round trip to discover ACAC on what was a simple request), couldn't the browser take the presence of ACAC on the response as a signal that it should have included credentials, and reissue the request? Of course it's not optimal, but it would remove the *need* to set withCredentials to get interoperability.
>
> It is not to mitigate a threat, at least not primarily.
>
> It is to avoid having to issue two requests. I think calling it an
> "optimization" is understating it a bit.
>
> Not only would issuing two requests many times likely be so slow that
> people would likely opt to use other solutions, such as JSONP.
>
> The other problem is that for requests that have side effects, issuing
> the request twice could of course be problematic. This applies to GET
> requests as well unfortunately since unfortunately they in practice
> often do have side effects.
>
> It's not clear to me how big this problem would be in practice though.
> For request that has side effects and that uses cookies, it is likely
> common that those cookies are used for authentication and thus issuing
> the request without cookies would result in a denied request and no
> side effects (maybe other than logging an error).
>
> So in that case issuing two requests, where the first one is without
> credentials, would be fine.
>
> But if cookies are used for other things than authentication, then
> issuing the request twice seems like it could easily be a problem.
>
> For that reason, I think re-issuing the request when an "ACAC: true"
> is returned would be bad (not to mention likely not web compatible at
> this point).
>
> All of this was debated while we did the initial CORS design.
>
> What I don't remember debating, and which might work, would be to add
> a different header which explicitly has the meaning "if this request
> was issued without credentials, please re-issue this request with
> credentials".
>
> Then leave it up to the server administrator to make sure that the
> initial, credential-less, request doesn't create side effects.
>
>
> I could see some privacy concerns with this though. Say that website A
> wants to use some REST API which website B provides. Currently website
> A can do this without exposing to B that the user is using website A.
> Simply issue CORS requests with .withCredentials is set to false.
>
> If we add a "please re-issue with credentials" header, then that means
> that website B can now figure out the users identity, assuming the
> user is logged in with B, by asking the request to be re-issued.
>
> Again, I don't know how big of a problem this would be.
>
>
> I'm also not sure how popular this feature would be with developers. I
> suspect most of them would prefer to use libraries which enable them
> to set .withCredentials to true, rather than pay the cost of issuing
> two requests.
>
> / Jonas

--
Mark Nottingham   https://www.mnot.net/


Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Tim Berners-Lee
In reply to this post by Jonas Sicking-2
Thank you Jonas, for that clarification.
some response inline.

> On 2016-01 -21, at 01:24, Jonas Sicking <[hidden email]> wrote:
>
> On Mon, Jan 18, 2016 at 3:27 PM, Mark Nottingham <[hidden email]> wrote:
>> ... or at least the motivations behind the decisions explained. It's pretty impenetrable now, and even security folks don't profess to know all of the details behind CORS any more.
>
> I'm bummed to hear that aspects of CORS is still confusing even to the
> TAG. This stuff likely needs to get documented someplace. I had hoped
> that it'd get documented in the spec, but maybe there's a better
> place?

Perhaps the problems are that it is complicated, rather arbitrary, and not derived from general principles..

>
> I'm not actually sure what exact confusion is being discussed in this
> thread is, so I'll address some of the questions I most commonly get.
>
> Q: What does the withCredentials flag do.
> A: When it's set to false, requests are sent containing only the
> information provided by the requesting website. I.e. the requesting
> website's provided URL, headers and request body. The only information
> that's added by the browser is information that's hardcoded into the
> browser and does not depend on user information. So for example the
> user-agent header. No cookies, authentication headers, or client-side
> certificates are added by the browser to the request before it is sent
> to the target website.
>
> However setting withCredentials to false does not prevent the
> requesting website from adding credentials through cookie headers,
> authentication headers, URL parameters or any other way that's exposed
> through the API which triggered the request.
>
> Additionally, the response data that would normally affect the client
> data storage is ignored. So for example set-cookie response headers
> are not written to the browsers cookie storage. The returned response
> is also not stored in the normal http cache, though if appropriate
> browsers may store it in a specific
> "CORS-requests-with-withCredentials-set-to-false" cache.
>
> When withCredentials is set to true, requests are handled like
> "normal" requests do in a browser. That means that cookies from the
> users cookie storage are added based on the target URL. Cached
> authentication data is added through the authentication header.
>
> The response is likewise processed like normal, so set-cookie headers
> are processed and the response is cached, if appropriate, in the
> normal browser http cache.
>
> Q: Why are security checks performed when withCredentials is set to false?
> A: Because the user, and the user's browser, might be behind a
> firewall and so might be able to access servers which a website would
> otherwise not be able to access.
>
> Sadly there is no, to me, known mechanism for detecting if a given
> server is behind a firewall.

That’s a long rathole but ...
1) If your local IP address is the same as the one you get from a public IP reflector then you are not behind a firewall
2) If your IP address starts with 192.168… then you are behind a firewall …  
3) BUT that isn’t the point, you can be outside a firewall and still have privileged access by your IP address.
4) And you could also be behind a carrier-grade NAT box but not have any privilege access as a result.

One possible but hard route is to pursue something like the router telling your machine whether it has no privileged access, which would then enable a lot of stuff.  So public internet spaces would set the flag, which would then mean the browsers would do less preflights, wasted attempts to access stuff, etc and so the browser would run more quickly for less bandwidth.

> Q: Is it safe to always set "Access-control-allow-origin: *" on all
> responses from a server.
> A: As long as the server is connected to the public internet, yes it
> is. It does not leak any information that couldn't be loaded using
> curl or any other non-browser HTTP client.

>
> If the server is behind a firewall and might contain sensitive
> information, the header should not be added.

Well, the header should be added for any public resource.
Some servers in fact handle the access control for the different resources on the site, and so in that case they ought to use that function to drive the headers automatically.  THAT is what should be coded up in the common servers.
That would help the server manager do the right thing.


>
> Q: Why does CORS not allow "Access-control-allow-origin: *" together
> with withCredentials=true?
> A: It was felt that this was too big of a foot gun.
>
> CORS was designed not long after Adobe had added the crossdomain.xml
> feature to Flash Player. The crossdomain.xml feature allows webserver
> administrators to easily indicate that the server contains resources
> that should be loadable from other origins. The feature only allowed
> "normal" requests, i.e. requests similar to ones that CORS makes when
> withCredentials=true.
>
> When crossdomain.xml was released many websites opted in allowing data
> to be read from other websites in order to share some public data that
> was hosted on the server. Unfortunately they forgot that some other
> URLs on the server served sensitive user data. The result was that
> relatively quickly after the release of the crossdomain.xml multiple
> websites leaked sensitive user data.
>
> You could argue that the problem was that crossdomain.xml was
> different since it is a per-server configuration file, whereas CORS
> uses per-URL headers. Hence CORS would be less prone to server
> administrators accidentally opting in to sharing on URLs that server
> user sensitive data.
>
> However in practice many (most?) popular web servers allow adding
> configuration files which add static http headers to all URLs under a
> given directory. So in practice on many servers it would have been
> just as easy to make the same mistake with CORS.

Any arguments about making things easy or difficult for server admins to
shoot themselves is protecting the (server) user from themselves which should always be done with care.
The user is the most important.
To first order, the system must implement a security protocol which allows
people to do the right thing — to give the right access to the right resources
by the right people and origins.  Yes, by all means allow the server to protect data
by default, but make it clear what is happening and allow the server operator easily to
tell the server what the situation is.  (you are/not running on the open internet.  This information itself is/not quite public)

When you make something impossible using HTTP in the browser, that is a big deal.

Q: Why was reflecting the incoming origin in the header the thing which was picked
as the way of saying “yes this really is public”?  Why not “access-control-allow-origin **” or something
 It is a pain to code, needs two or three lines of not-newbie-obvious .htaccess in Apache, etc.
Result? the recipe for fixing it is sent around to do the origin reflection, and new server code does it by default for everything.
Because CORS is such a pain for developers to deal with on the client side, with no error codes, etc
that servers who want stuff to just work, and slap in the strongest CORS medicine they find on the net.



> Q: Why does CORS not allow listing multiple origins, or allow pattern
> matching, in the "Access-control-allow-origin" header?
> A: It was felt that if the server uses dynamic server-side logic to
> generate responses for a given URL, that they could also then
> dynamically generate the appropriate Access-control-allow-origin
> header.
> For servers that generate static responses you can generally simply
> use "Access-control-allow-origin: *”.

Well no, not if they only want 7 specific domains to have access.

> Keep in mind that static
> responses can generally be read from non-browser HTTP clients like
> curl anyway.
>
> This doesn't account for static responses which are password protected
> using either cookies or auth headers. So yeah, our solution here is
> not perfect, but we decided to opt for simplicity.
>
> My personal hope was also that generic server modules would be written
> to handle CORS support and which would simplify situations like this.
> I'm not sure if such modules exist yet or not.

There are lots. They may be turned on by default.  A concern is they tend to just defeat CORS
and they don’t necessarily distinguish between public resources and others.

Also people use CORS proxies to access the web, which are associated


> [...]
>
>
> If I'm not addressing the concern/questions from the TAG then please
> let me know.

I think the top three issues the TAG had were basically

a) Having the withCredentials flag as a parameter to fetch() is broken.  In general the middleware which calls fetch() will not have magic application-level knowledge of which resources it is going to fetch are public, which are private.   So a general the fetch has to work without that hint, and do the right thing. (opinions vary here)

b) For a webapp which needs to load stuff from the net, the lack of clear error conditions makes it hard to understand what is going on.

c) Asking server writers to do the origin reflection thing is unreasonable


>
> I'd really love it if this type of information could make it into the
> spec in a way that is understandable to more people.
>
> / Jonas
>


Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Jonas Sicking-2
>> Q: Why are security checks performed when withCredentials is set to false?
>> A: Because the user, and the user's browser, might be behind a
>> firewall and so might be able to access servers which a website would
>> otherwise not be able to access.
>>
>> Sadly there is no, to me, known mechanism for detecting if a given
>> server is behind a firewall.
>
> That’s a long rathole but ...
> 1) If your local IP address is the same as the one you get from a public IP reflector then you are not behind a firewall
> 2) If your IP address starts with 192.168… then you are behind a firewall …
> 3) BUT that isn’t the point, you can be outside a firewall and still have privileged access by your IP address.
> 4) And you could also be behind a carrier-grade NAT box but not have any privilege access as a result.
>
> One possible but hard route is to pursue something like the router telling your machine whether it has no privileged access, which would then enable a lot of stuff.  So public internet spaces would set the flag, which would then mean the browsers would do less preflights, wasted attempts to access stuff, etc and so the browser would run more quickly for less bandwidth.

If we get to a point where clients can reliably tell if a given
request goes to a server that is behind a firewall or not, then we
should certainly take advantage of that.

>> Q: Why does CORS not allow "Access-control-allow-origin: *" together
>> with withCredentials=true?
>> A: It was felt that this was too big of a foot gun.
>>
>> CORS was designed not long after Adobe had added the crossdomain.xml
>> feature to Flash Player. The crossdomain.xml feature allows webserver
>> administrators to easily indicate that the server contains resources
>> that should be loadable from other origins. The feature only allowed
>> "normal" requests, i.e. requests similar to ones that CORS makes when
>> withCredentials=true.
>>
>> When crossdomain.xml was released many websites opted in allowing data
>> to be read from other websites in order to share some public data that
>> was hosted on the server. Unfortunately they forgot that some other
>> URLs on the server served sensitive user data. The result was that
>> relatively quickly after the release of the crossdomain.xml multiple
>> websites leaked sensitive user data.
>>
>> You could argue that the problem was that crossdomain.xml was
>> different since it is a per-server configuration file, whereas CORS
>> uses per-URL headers. Hence CORS would be less prone to server
>> administrators accidentally opting in to sharing on URLs that server
>> user sensitive data.
>>
>> However in practice many (most?) popular web servers allow adding
>> configuration files which add static http headers to all URLs under a
>> given directory. So in practice on many servers it would have been
>> just as easy to make the same mistake with CORS.
>
> Any arguments about making things easy or difficult for server admins to
> shoot themselves is protecting the (server) user from themselves which should always be done with care.
> The user is the most important.

Absolutely. The current CORS design was done after a lot of web
administrators which deployed crossdomain.xml did so in a manner that
leaked sensitive user data to anyone with a website.

> To first order, the system must implement a security protocol which allows
> people to do the right thing — to give the right access to the right resources
> by the right people and origins.  Yes, by all means allow the server to protect data
> by default, but make it clear what is happening and allow the server operator easily to
> tell the server what the situation is.  (you are/not running on the open internet.  This information itself is/not quite public)
>
> When you make something impossible using HTTP in the browser, that is a big deal.

Yup. Though as far as I can tell, none of the use cases discussed so
far is impossible with CORS. In fact, we worked quite hard to make all
of HTTP accessible in even the initial release of CORS, despite the
fact the by far most common use cases only use a small subset of HTTP.

> Q: Why was reflecting the incoming origin in the header the thing which was picked
> as the way of saying “yes this really is public”?  Why not “access-control-allow-origin **” or something
>  It is a pain to code, needs two or three lines of not-newbie-obvious .htaccess in Apache, etc.

The main reason that reflecting the origin was chosen was so that an
intermediate cache wouldn't cause the browser to see the signal "yes
this is a public resource".

> Result? the recipe for fixing it is sent around to do the origin reflection, and new server code does it by default for everything.
> Because CORS is such a pain for developers to deal with on the client side, with no error codes, etc
> that servers who want stuff to just work, and slap in the strongest CORS medicine they find on the net.

I've never heard of servers doing this origin-reflection by default.
If that really is the case then I agree that that would be very
concerning since it would put us in an even worse state than
crossdomain.xml was in.

>> Q: Why does CORS not allow listing multiple origins, or allow pattern
>> matching, in the "Access-control-allow-origin" header?
>> A: It was felt that if the server uses dynamic server-side logic to
>> generate responses for a given URL, that they could also then
>> dynamically generate the appropriate Access-control-allow-origin
>> header.
>> For servers that generate static responses you can generally simply
>> use "Access-control-allow-origin: *”.
>
> Well no, not if they only want 7 specific domains to have access.

If the response is entirely static, and the server is connected to the
public internet, then there is little benefit to limiting the response
to 7 specific domains.

In practice such a response is world readable anyway since anyone in
the world could use curl or wget to read the data. No matter what is
sent in any CORS headers.

>> Keep in mind that static
>> responses can generally be read from non-browser HTTP clients like
>> curl anyway.
>>
>> This doesn't account for static responses which are password protected
>> using either cookies or auth headers. So yeah, our solution here is
>> not perfect, but we decided to opt for simplicity.
>>
>> My personal hope was also that generic server modules would be written
>> to handle CORS support and which would simplify situations like this.
>> I'm not sure if such modules exist yet or not.
>
> There are lots. They may be turned on by default.  A concern is they tend to just defeat CORS
> and they don’t necessarily distinguish between public resources and others.
>
> Also people use CORS proxies to access the web, which are associated

Does anyone have any examples of websites that have shared data that
should have been kept private publicly?

We saw plenty of examples where that happened with crossdomain.xml,
but I've personally not heard that it's happened with CORS. But
absence of evidence is not evidence of absence, so I'd be very
interested to hear about any breaches.


> I think the top three issues the TAG had were basically
>
> a) Having the withCredentials flag as a parameter to fetch() is broken.  In general the middleware which calls fetch() will not have magic application-level knowledge of which resources it is going to fetch are public, which are private.   So a general the fetch has to work without that hint, and do the right thing. (opinions vary here)

As was requested in
https://github.com/w3ctag/spec-reviews/issues/76#issuecomment-183317897
CORS enables a mode where you don't have to have any knowledge about
what the security policy of the server is, and you only have to define
a URL. CORS then adapts the wire protocol in order to make that safe.

The way that you do this is that you always set the withCredentials
flag to true. Then, as wire protocol you send the
access-control-with-credentials and access-control-allow-origin
headers with the appropriate values.

> b) For a webapp which needs to load stuff from the net, the lack of clear error conditions makes it hard to understand what is going on.
> c) Asking server writers to do the origin reflection thing is unreasonable

I suggest that the TAG work with the webappsec WG in order to come up
with concrete proposals for these two. That way we can compare
security aspects, wire protocols, performance, etc.

/ Jonas

Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

L. David Baron
In reply to this post by Tim Berners-Lee
On Thursday 2016-03-31 16:51 +0100, Tim Berners-Lee wrote:
> Q: Why was reflecting the incoming origin in the header the thing which was picked
> as the way of saying “yes this really is public”?  Why not “access-control-allow-origin **” or something
>  It is a pain to code, needs two or three lines of not-newbie-obvious .htaccess in Apache, etc.
> Result? the recipe for fixing it is sent around to do the origin reflection, and new server code does it by default for everything.
> Because CORS is such a pain for developers to deal with on the client side, with no error codes, etc
> that servers who want stuff to just work, and slap in the strongest CORS medicine they find on the net.

One important point here that I failed to realize during the meeting
is that origin reflection is required only in a particularly unusual
case:  for a resource that both:

 (a) requires credentials (e.g., cookies, HTTP auth) to access

 (b) is public.

I think this case ought to be unusual since public resources
generally shouldn't require credentials to access.

A far easier solution for the server administrator in this case
should be allowing access to the resource without credentials.
(This is what doing origin reflection would do anyway, through
browser-based attacks on the people who do have credentials.)

> I think the top three issues the TAG had were basically
>
> a) Having the withCredentials flag as a parameter to fetch() is broken.  In general the middleware which calls fetch() will not have magic application-level knowledge of which resources it is going to fetch are public, which are private.   So a general the fetch has to work without that hint, and do the right thing. (opinions vary here)

I disagree (as I said), since I think the two options the flag has
today should exist.  They represent behaviors that exist in the Web
platform and that ought to be exposed.  That doesn't necessarily
mean that there shouldn't be other options if such options are also
needed.

> b) For a webapp which needs to load stuff from the net, the lack of clear error conditions makes it hard to understand what is going on.

I agree this is a problem.

(There may be security risk in reporting particular error details to
the calling script, although I don't know if that's the case here.
But even if it is, it should be possible to report clear errors to
the console.)

> c) Asking server writers to do the origin reflection thing is unreasonable

Given my comments above (unless I'm mistaken), I think I now
disagree with this point as well.

-David

--
𝄞   L. David Baron                         http://dbaron.org/   𝄂
𝄢   Mozilla                          https://www.mozilla.org/   𝄂
             Before I built a wall I'd ask to know
             What I was walling in or walling out,
             And to whom I was like to give offense.
               - Robert Frost, Mending Wall (1914)

signature.asc (836 bytes) Download Attachment
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Tim Berners-Lee
In reply to this post by Jonas Sicking-2

> On 2016-03 -31, at 18:50, Jonas Sicking <[hidden email]> wrote:
>
>>> Q: Why are security checks performed when withCredentials is set to false?
>>> A: Because the user, and the user's browser, might be behind a
>>> firewall and so might be able to access servers which a website would
>>> otherwise not be able to access.
>>>
>>> Sadly there is no, to me, known mechanism for detecting if a given
>>> server is behind a firewall.
>>
>> That’s a long rathole but ...
>> 1) If your local IP address is the same as the one you get from a public IP reflector then you are not behind a firewall
>> 2) If your IP address starts with 192.168… then you are behind a firewall …
>> 3) BUT that isn’t the point, you can be outside a firewall and still have privileged access by your IP address.
>> 4) And you could also be behind a carrier-grade NAT box but not have any privilege access as a result.
>>
>> One possible but hard route is to pursue something like the router telling your machine whether it has no privileged access, which would then enable a lot of stuff.  So public internet spaces would set the flag, which would then mean the browsers would do less preflights, wasted attempts to access stuff, etc and so the browser would run more quickly for less bandwidth.
>
> If we get to a point where clients can reliably tell if a given
> request goes to a server that is behind a firewall or not, then we
> should certainly take advantage of that.

yes


>
>>> Q: Why does CORS not allow "Access-control-allow-origin: *" together
>>> with withCredentials=true?
>>> A: It was felt that this was too big of a foot gun.
>>>
>>> CORS was designed not long after Adobe had added the crossdomain.xml
>>> feature to Flash Player. The crossdomain.xml feature allows webserver
>>> administrators to easily indicate that the server contains resources
>>> that should be loadable from other origins. The feature only allowed
>>> "normal" requests, i.e. requests similar to ones that CORS makes when
>>> withCredentials=true.
>>>
>>> When crossdomain.xml was released many websites opted in allowing data
>>> to be read from other websites in order to share some public data that
>>> was hosted on the server. Unfortunately they forgot that some other
>>> URLs on the server served sensitive user data. The result was that
>>> relatively quickly after the release of the crossdomain.xml multiple
>>> websites leaked sensitive user data.
>>>
>>> You could argue that the problem was that crossdomain.xml was
>>> different since it is a per-server configuration file, whereas CORS
>>> uses per-URL headers. Hence CORS would be less prone to server
>>> administrators accidentally opting in to sharing on URLs that server
>>> user sensitive data.
>>>
>>> However in practice many (most?) popular web servers allow adding
>>> configuration files which add static http headers to all URLs under a
>>> given directory. So in practice on many servers it would have been
>>> just as easy to make the same mistake with CORS.
>>
>> Any arguments about making things easy or difficult for server admins to
>> shoot themselves is protecting the (server) user from themselves which should always be done with care.
>> The user is the most important.
>
> Absolutely. The current CORS design was done after a lot of web
> administrators which deployed crossdomain.xml did so in a manner that
> leaked sensitive user data to anyone with a website.

>
>> To first order, the system must implement a security protocol which allows
>> people to do the right thing — to give the right access to the right resources
>> by the right people and origins.  Yes, by all means allow the server to protect data
>> by default, but make it clear what is happening and allow the server operator easily to
>> tell the server what the situation is.  (you are/not running on the open internet.  This information itself is/not quite public)
>>
>> When you make something impossible using HTTP in the browser, that is a big deal.
>
> Yup. Though as far as I can tell, none of the use cases discussed so
> far is impossible with CORS. In fact, we worked quite hard to make all
> of HTTP accessible in even the initial release of CORS, despite the
> fact the by far most common use cases only use a small subset of HTTP.
>
>> Q: Why was reflecting the incoming origin in the header the thing which was picked
>> as the way of saying “yes this really is public”?  Why not “access-control-allow-origin **” or something
>> It is a pain to code, needs two or three lines of not-newbie-obvious .htaccess in Apache, etc.
>
> The main reason that reflecting the origin was chosen was so that an
> intermediate cache wouldn't cause the browser to see the signal "yes
> this is a public resource”.

This is the first time I have come across this reason. Interesting.
What was the failure mode?   The web browser goes though an intermediate caching proxy.
The proxy either forwards all the headers both ways, then surely everything works, with * or reflection of the origin?
What was the failure mode you were thinking of?

>
>> Result? the recipe for fixing it is sent around to do the origin reflection, and new server code does it by default for everything.
>> Because CORS is such a pain for developers to deal with on the client side, with no error codes, etc
>> that servers who want stuff to just work, and slap in the strongest CORS medicine they find on the net.
>
> I've never heard of servers doing this origin-reflection by default.
> If that really is the case then I agree that that would be very
> concerning since it would put us in an even worse state than
> crossdomain.xml was in.

Yes

>>> Q: Why does CORS not allow listing multiple origins, or allow pattern
>>> matching, in the "Access-control-allow-origin" header?
>>> A: It was felt that if the server uses dynamic server-side logic to
>>> generate responses for a given URL, that they could also then
>>> dynamically generate the appropriate Access-control-allow-origin
>>> header.
>>> For servers that generate static responses you can generally simply
>>> use "Access-control-allow-origin: *”.
>>
>> Well no, not if they only want 7 specific domains to have access.
>
> If the response is entirely static, and the server is connected to the
> public internet, then there is little benefit to limiting the response
> to 7 specific domains.
>
> In practice such a response is world readable anyway since anyone in
> the world could use curl or wget to read the data. No matter what is
> sent in any CORS headers.

You are right that there is no point allowing any web page to access something if it is not public.

Suppose the resource is not public.   It is only available to certain people.
Imagine that we had a list of w3c C reps and their phone numbers which
we wanted to be accessible to those reps (using their passwords)
and we wanted them to be able to use web apps on tools.ietf.org and
tools.w3.org to access them, and no other web apps to access that data.

The server checks the authorization of the user and rejects it 401 or 403 if not OK.
It then checks the origin header and if it does not match the two origins it gives a 499 error code which we don’t have???
There is no point returning data to an app which isn’t allowed it.

If the origin, say tools.ietf.org, (is omitted or) matches, then it returns 200 and could give both origins in the ACAO header,  but it can’t it has to reflect the current requesting origin.  
(If that response is cached by a proxy, which later respond s to a request from tools.w3.org. But presumably the server sends Vary:Origin to prevent that happening in the proxy)




>
>>> Keep in mind that static
>>> responses can generally be read from non-browser HTTP clients like
>>> curl anyway.
>>>
>>> This doesn't account for static responses which are password protected
>>> using either cookies or auth headers. So yeah, our solution here is
>>> not perfect, but we decided to opt for simplicity.
>>>
>>> My personal hope was also that generic server modules would be written
>>> to handle CORS support and which would simplify situations like this.
>>> I'm not sure if such modules exist yet or not.
>>
>> There are lots. They may be turned on by default.  A concern is they tend to just defeat CORS
>> and they don’t necessarily distinguish between public resources and others.
>>
>> Also people use CORS proxies to access the web, which are associated
>
> Does anyone have any examples of websites that have shared data that
> should have been kept private publicly?
>
> We saw plenty of examples where that happened with crossdomain.xml,
> but I've personally not heard that it's happened with CORS. But
> absence of evidence is not evidence of absence, so I'd be very
> interested to hear about any breaches.
>
>
>> I think the top three issues the TAG had were basically
>>
>> a) Having the withCredentials flag as a parameter to fetch() is broken.  In general the middleware which calls fetch() will not have magic application-level knowledge of which resources it is going to fetch are public, which are private.   So a general the fetch has to work without that hint, and do the right thing. (opinions vary here)
>
> As was requested in
> https://github.com/w3ctag/spec-reviews/issues/76#issuecomment-183317897
> CORS enables a mode where you don't have to have any knowledge about
> what the security policy of the server is, and you only have to define
> a URL. CORS then adapts the wire protocol in order to make that safe.
>
> The way that you do this is that you always set the withCredentials
> flag to true. Then, as wire protocol you send the
> access-control-with-credentials and access-control-allow-origin
> headers with the appropriate values.

Ok, so then the protocol is that the withCredentials flag is always set in the client.
Then the “ACAC: * ”  header is never accepted by the browser, and so logically it should be never sent by the server, and removed from the documentation.  The server returns 200  with ACAC <origin>  if the document is public or the user is allowed access. The browser has no way of knowing whether the resource is actually public or not, whether it can be shard with specific other origins, or with the whole world.  That surely seems less than optimal, disabling useful caching?



(Maybe we should go back to the server response like
        Public: GET
to be able to just flag that something is not protected at all.)


>
>> b) For a webapp which needs to load stuff from the net, the lack of clear error conditions makes it hard to understand what is going on.
>> c) Asking server writers to do the origin reflection thing is unreasonable
>
> I suggest that the TAG work with the webappsec WG in order to come up
> with concrete proposals for these two. That way we can compare
> security aspects, wire protocols, performance, etc.

Sounds like a good idea.

>
> / Jonas
>


Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Tim Berners-Lee
In reply to this post by Jonas Sicking-2
Thank you Jonas, for that clarification.
some response inline.

> On 2016-01 -21, at 01:24, Jonas Sicking <[hidden email]> wrote:
>
> On Mon, Jan 18, 2016 at 3:27 PM, Mark Nottingham <[hidden email]> wrote:
>> ... or at least the motivations behind the decisions explained. It's pretty impenetrable now, and even security folks don't profess to know all of the details behind CORS any more.
>
> I'm bummed to hear that aspects of CORS is still confusing even to the
> TAG. This stuff likely needs to get documented someplace. I had hoped
> that it'd get documented in the spec, but maybe there's a better
> place?

Perhaps the problems are that it is complicated, rather arbitrary, and not derived from general principles..

>
> I'm not actually sure what exact confusion is being discussed in this
> thread is, so I'll address some of the questions I most commonly get.
>
> Q: What does the withCredentials flag do.
> A: When it's set to false, requests are sent containing only the
> information provided by the requesting website. I.e. the requesting
> website's provided URL, headers and request body. The only information
> that's added by the browser is information that's hardcoded into the
> browser and does not depend on user information. So for example the
> user-agent header. No cookies, authentication headers, or client-side
> certificates are added by the browser to the request before it is sent
> to the target website.
>
> However setting withCredentials to false does not prevent the
> requesting website from adding credentials through cookie headers,
> authentication headers, URL parameters or any other way that's exposed
> through the API which triggered the request.
>
> Additionally, the response data that would normally affect the client
> data storage is ignored. So for example set-cookie response headers
> are not written to the browsers cookie storage. The returned response
> is also not stored in the normal http cache, though if appropriate
> browsers may store it in a specific
> "CORS-requests-with-withCredentials-set-to-false" cache.
>
> When withCredentials is set to true, requests are handled like
> "normal" requests do in a browser. That means that cookies from the
> users cookie storage are added based on the target URL. Cached
> authentication data is added through the authentication header.
>
> The response is likewise processed like normal, so set-cookie headers
> are processed and the response is cached, if appropriate, in the
> normal browser http cache.
>
> Q: Why are security checks performed when withCredentials is set to false?
> A: Because the user, and the user's browser, might be behind a
> firewall and so might be able to access servers which a website would
> otherwise not be able to access.
>
> Sadly there is no, to me, known mechanism for detecting if a given
> server is behind a firewall.

That’s a long rathole but ...
1) If your local IP address is the same as the one you get from a public IP reflector then you are not behind a firewall
2) If your IP address starts with 192.168… then you are behind a firewall …  
3) BUT that isn’t the point, you can be outside a firewall and still have privileged access by your IP address.
4) And you could also be behind a carrier-grade NAT box but not have any privilege access as a result.

One possible but hard route is to pursue something like the router telling your machine whether it has no privileged access, which would then enable a lot of stuff.  So public internet spaces would set the flag, which would then mean the browsers would do less preflights, wasted attempts to access stuff, etc and so the browser would run more quickly for less bandwidth.

> Q: Is it safe to always set "Access-control-allow-origin: *" on all
> responses from a server.
> A: As long as the server is connected to the public internet, yes it
> is. It does not leak any information that couldn't be loaded using
> curl or any other non-browser HTTP client.

>
> If the server is behind a firewall and might contain sensitive
> information, the header should not be added.

Well, the header should be added for any public resource.
Some servers in fact handle the access control for the different resources on the site, and so in that case they ought to use that function to drive the headers automatically.  THAT is what should be coded up in the common servers.
That would help the server manager do the right thing.


>
> Q: Why does CORS not allow "Access-control-allow-origin: *" together
> with withCredentials=true?
> A: It was felt that this was too big of a foot gun.
>
> CORS was designed not long after Adobe had added the crossdomain.xml
> feature to Flash Player. The crossdomain.xml feature allows webserver
> administrators to easily indicate that the server contains resources
> that should be loadable from other origins. The feature only allowed
> "normal" requests, i.e. requests similar to ones that CORS makes when
> withCredentials=true.
>
> When crossdomain.xml was released many websites opted in allowing data
> to be read from other websites in order to share some public data that
> was hosted on the server. Unfortunately they forgot that some other
> URLs on the server served sensitive user data. The result was that
> relatively quickly after the release of the crossdomain.xml multiple
> websites leaked sensitive user data.
>
> You could argue that the problem was that crossdomain.xml was
> different since it is a per-server configuration file, whereas CORS
> uses per-URL headers. Hence CORS would be less prone to server
> administrators accidentally opting in to sharing on URLs that server
> user sensitive data.
>
> However in practice many (most?) popular web servers allow adding
> configuration files which add static http headers to all URLs under a
> given directory. So in practice on many servers it would have been
> just as easy to make the same mistake with CORS.

Any arguments about making things easy or difficult for server admins to
shoot themselves in the foot coming from a non-optimal attitude.
To first order, the system must implement a security protocol which allows
people to do the right thing — to give the right access to the right resources
by the right people and origins.  Yes, by all means make the server

Q: Why was reflecting the incoming origin in the header the thing which was picked
as the ay of saying “yes this really is public”?  Why not “access-control-allow-origin **” or something
 It is a pain to code, needs two or three lines of not-newbie-obvious .htaccess in Apache, etc.
Result? the recipe is sent around
and new server code does it by default for everything.

Because CORS is such a pain for developers to deal with on the client side, with no error codes, etc
that servers who want stuff to just work, and slap in the strongest CORS medicine they find on the net.



> Q: Why does CORS not allow listing multiple origins, or allow pattern
> matching, in the "Access-control-allow-origin" header?
> A: It was felt that if the server uses dynamic server-side logic to
> generate responses for a given URL, that they could also then
> dynamically generate the appropriate Access-control-allow-origin
> header.
> For servers that generate static responses you can generally simply
> use "Access-control-allow-origin: *”.

Well no, not if they only want 7 specific domains to have access.

> Keep in mind that static
> responses can generally be read from non-browser HTTP clients like
> curl anyway.
>
> This doesn't account for static responses which are password protected
> using either cookies or auth headers. So yeah, our solution here is
> not perfect, but we decided to opt for simplicity.
>
> My personal hope was also that generic server modules would be written
> to handle CORS support and which would simplify situations like this.
> I'm not sure if such modules exist yet or not.

There are lots. They may be turned on by default.  A concern is they tend to just defeat CORS
and they don’t necessarily distinguish between public resources and others.

Also people use CORS proxies to access the web, which are associated


> [...]
>
>
> If I'm not addressing the concern/questions from the TAG then please
> let me know.

I think the top two issues the TAG had is

a) Having the withCredentials flag as a parameter to fetch() is broken.  In general the middleware which calls fetch() will not have magic application-level knowledge of which resources it is going to fetch are public, which are private.   So a general the fetch has to work without that hint, and do the right thing.

b) For a webapp which needs to load stuff from the net, the lack of clear error conditions makes it hard to understand what is going on.    A good

c) Asking server writers to do the origin reflection thing is unreasonable


>
> I'd really love it if this type of information could make it into the
> spec in a way that is understandable to more people.
>
> / Jonas
>


Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Henry Story-4

> On 1 Apr 2016, at 19:26, Tim Berners-Lee <[hidden email]> wrote:
>
>>
>> On 2016-01 -21, at 01:24, Jonas Sicking <[hidden email]> wrote:
>>
>> On Mon, Jan 18, 2016 at 3:27 PM, Mark Nottingham <[hidden email]> wrote:
>>> ... or at least the motivations behind the decisions explained. It's pretty impenetrable now, and even security folks don't profess to know all of the details behind CORS any more.
>>
>> I'm bummed to hear that aspects of CORS is still confusing even to the
>> TAG. This stuff likely needs to get documented someplace. I had hoped
>> that it'd get documented in the spec, but maybe there's a better
>> place?
>
> Perhaps the problems are that it is complicated, rather arbitrary, and not derived from general principles..

My guess is that to build this on general principles it would be useful to work with
some form of modal logic, eg. doxastic logic [1]. Perhaps this has allready been
done. We have a number of actors that are part of the protocol:

 a. the origin agent O
 b. the browser B
 c. the web server S
 d. the resource R

The browser needs to tell the web server that a request is not coming from it, but
that it is relaying the information from a JS actor named by the origin of the JS.

So something like B is relaying that O wants access X to resource R.

This could be formalised and then the reasons of the decisions might become clearer.
It took me quite a lot of thinking before it became clear what some of the reasons
for some of the decisions might be...

Henry

[1] https://en.wikipedia.org/wiki/Doxastic_logic
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Comment on minutes ## With Credentials flag etc

Brad Hill
In reply to this post by Tim Berners-Lee
I've (at last) attempted to write a developer-friendly narrative explanation of all of this stuff here.  How CORS works, the permission model, and some of the historical reasoning behind the various choices.

https://docs.google.com/document/d/1AtxTDw-g9BSRW9n9kGTTqNkDTGcVfSKPAOjVGkPFu2k/edit?usp=sharing

Yes, it's a Google doc for now since that was fastest for me to compose, but if people think it is useful, I can incorporate comments and maybe we can turn it into a WebAppSec WG note or joint TAG finding.

-Brad



On 4/1/16, 11:26 AM, "Tim Berners-Lee" <[hidden email]> wrote:

>Thank you Jonas, for that clarification.
>some response inline.
>
>> On 2016-01 -21, at 01:24, Jonas Sicking <[hidden email]> wrote:
>>
>> On Mon, Jan 18, 2016 at 3:27 PM, Mark Nottingham <[hidden email]> wrote:
>>> ... or at least the motivations behind the decisions explained. It's pretty impenetrable now, and even security folks don't profess to know all of the details behind CORS any more.
>>
>> I'm bummed to hear that aspects of CORS is still confusing even to the
>> TAG. This stuff likely needs to get documented someplace. I had hoped
>> that it'd get documented in the spec, but maybe there's a better
>> place?
>
>Perhaps the problems are that it is complicated, rather arbitrary, and not derived from general principles..
>
>>
>> I'm not actually sure what exact confusion is being discussed in this
>> thread is, so I'll address some of the questions I most commonly get.
>>
>> Q: What does the withCredentials flag do.
>> A: When it's set to false, requests are sent containing only the
>> information provided by the requesting website. I.e. the requesting
>> website's provided URL, headers and request body. The only information
>> that's added by the browser is information that's hardcoded into the
>> browser and does not depend on user information. So for example the
>> user-agent header. No cookies, authentication headers, or client-side
>> certificates are added by the browser to the request before it is sent
>> to the target website.
>>
>> However setting withCredentials to false does not prevent the
>> requesting website from adding credentials through cookie headers,
>> authentication headers, URL parameters or any other way that's exposed
>> through the API which triggered the request.
>>
>> Additionally, the response data that would normally affect the client
>> data storage is ignored. So for example set-cookie response headers
>> are not written to the browsers cookie storage. The returned response
>> is also not stored in the normal http cache, though if appropriate
>> browsers may store it in a specific
>> "CORS-requests-with-withCredentials-set-to-false" cache.
>>
>> When withCredentials is set to true, requests are handled like
>> "normal" requests do in a browser. That means that cookies from the
>> users cookie storage are added based on the target URL. Cached
>> authentication data is added through the authentication header.
>>
>> The response is likewise processed like normal, so set-cookie headers
>> are processed and the response is cached, if appropriate, in the
>> normal browser http cache.
>>
>> Q: Why are security checks performed when withCredentials is set to false?
>> A: Because the user, and the user's browser, might be behind a
>> firewall and so might be able to access servers which a website would
>> otherwise not be able to access.
>>
>> Sadly there is no, to me, known mechanism for detecting if a given
>> server is behind a firewall.
>
>That’s a long rathole but ...
>1) If your local IP address is the same as the one you get from a public IP reflector then you are not behind a firewall
>2) If your IP address starts with 192.168… then you are behind a firewall …  
>3) BUT that isn’t the point, you can be outside a firewall and still have privileged access by your IP address.
>4) And you could also be behind a carrier-grade NAT box but not have any privilege access as a result.
>
>One possible but hard route is to pursue something like the router telling your machine whether it has no privileged access, which would then enable a lot of stuff.  So public internet spaces would set the flag, which would then mean the browsers would do less preflights, wasted attempts to access stuff, etc and so the browser would run more quickly for less bandwidth.
>
>> Q: Is it safe to always set "Access-control-allow-origin: *" on all
>> responses from a server.
>> A: As long as the server is connected to the public internet, yes it
>> is. It does not leak any information that couldn't be loaded using
>> curl or any other non-browser HTTP client.
>
>>
>> If the server is behind a firewall and might contain sensitive
>> information, the header should not be added.
>
>Well, the header should be added for any public resource.
>Some servers in fact handle the access control for the different resources on the site, and so in that case they ought to use that function to drive the headers automatically.  THAT is what should be coded up in the common servers.
>That would help the server manager do the right thing.
>
>
>>
>> Q: Why does CORS not allow "Access-control-allow-origin: *" together
>> with withCredentials=true?
>> A: It was felt that this was too big of a foot gun.
>>
>> CORS was designed not long after Adobe had added the crossdomain.xml
>> feature to Flash Player. The crossdomain.xml feature allows webserver
>> administrators to easily indicate that the server contains resources
>> that should be loadable from other origins. The feature only allowed
>> "normal" requests, i.e. requests similar to ones that CORS makes when
>> withCredentials=true.
>>
>> When crossdomain.xml was released many websites opted in allowing data
>> to be read from other websites in order to share some public data that
>> was hosted on the server. Unfortunately they forgot that some other
>> URLs on the server served sensitive user data. The result was that
>> relatively quickly after the release of the crossdomain.xml multiple
>> websites leaked sensitive user data.
>>
>> You could argue that the problem was that crossdomain.xml was
>> different since it is a per-server configuration file, whereas CORS
>> uses per-URL headers. Hence CORS would be less prone to server
>> administrators accidentally opting in to sharing on URLs that server
>> user sensitive data.
>>
>> However in practice many (most?) popular web servers allow adding
>> configuration files which add static http headers to all URLs under a
>> given directory. So in practice on many servers it would have been
>> just as easy to make the same mistake with CORS.
>
>Any arguments about making things easy or difficult for server admins to
>shoot themselves in the foot coming from a non-optimal attitude.
>To first order, the system must implement a security protocol which allows
>people to do the right thing — to give the right access to the right resources
>by the right people and origins.  Yes, by all means make the server
>
>Q: Why was reflecting the incoming origin in the header the thing which was picked
>as the ay of saying “yes this really is public”?  Why not “access-control-allow-origin **” or something
> It is a pain to code, needs two or three lines of not-newbie-obvious .htaccess in Apache, etc.
>Result? the recipe is sent around
>and new server code does it by default for everything.
>
>Because CORS is such a pain for developers to deal with on the client side, with no error codes, etc
>that servers who want stuff to just work, and slap in the strongest CORS medicine they find on the net.
>
>
>
>> Q: Why does CORS not allow listing multiple origins, or allow pattern
>> matching, in the "Access-control-allow-origin" header?
>> A: It was felt that if the server uses dynamic server-side logic to
>> generate responses for a given URL, that they could also then
>> dynamically generate the appropriate Access-control-allow-origin
>> header.
>> For servers that generate static responses you can generally simply
>> use "Access-control-allow-origin: *”.
>
>Well no, not if they only want 7 specific domains to have access.
>
>> Keep in mind that static
>> responses can generally be read from non-browser HTTP clients like
>> curl anyway.
>>
>> This doesn't account for static responses which are password protected
>> using either cookies or auth headers. So yeah, our solution here is
>> not perfect, but we decided to opt for simplicity.
>>
>> My personal hope was also that generic server modules would be written
>> to handle CORS support and which would simplify situations like this.
>> I'm not sure if such modules exist yet or not.
>
>There are lots. They may be turned on by default.  A concern is they tend to just defeat CORS
>and they don’t necessarily distinguish between public resources and others.
>
>Also people use CORS proxies to access the web, which are associated
>
>
>> [...]
>>
>>
>> If I'm not addressing the concern/questions from the TAG then please
>> let me know.
>
>I think the top two issues the TAG had is
>
>a) Having the withCredentials flag as a parameter to fetch() is broken.  In general the middleware which calls fetch() will not have magic application-level knowledge of which resources it is going to fetch are public, which are private.   So a general the fetch has to work without that hint, and do the right thing.
>
>b) For a webapp which needs to load stuff from the net, the lack of clear error conditions makes it hard to understand what is going on.    A good
>
>c) Asking server writers to do the origin reflection thing is unreasonable
>
>
>>
>> I'd really love it if this type of information could make it into the
>> spec in a way that is understandable to more people.
>>
>> / Jonas
>>
>
>
12
Loading...