GM_xmlhttpRequest referer not passing?

Hey everyone,
I've been working on a little script that has to send a request to a specific page using xmlhttpreq, and while it seems to be sending everything else, it is not sending the referer header information. I have verified this with packet sniffing.

I've done extensive googling but have not found a specific answer, it almost seems like firefox is dropping the referer header tag? What's the deal?
Is there a way around it?

No, GM_xmlhttpRequest does not send a referer header (or it is always blank). I assume this is a bug in Greasemonkey, or perhaps it's intentional. I tried searching for info on it but gave up. I did not see it listed here: http://greasemonkey.mozdev.org/bugs.html

I have posted a bug request here: https://www.mozdev.org/bugs/show_bug.cgi?id=18104

It really was a key part of a test script I was making, quite annoying.
Can you think of any workarounds?

I haven't tried just using XMLHttpRequest, maybe it works different, or maybe it does the same thing, it's worth a try.

Another thing you might want to know: when using GM_xmlhttpRequest the browser caches the requests (at least SeaMonkey does). The first request shows up in the logs but successive requests do not until the browser cache is emptied, then the first request will show again. Normally I would see a 304 (page not modified), but I didn't even see this. I even tried setting headers for caching, which I forget now, but it wouldn't matter if the browser is just pulling it out of the cache.
I know this doesn't help in sending a referrer, just thought you'd like to know.

XMLHttpRequest passes Referer header correctly.
The workaround for GM is making request to "http://your_server.com/your_script.php?par=" + encodeURIComponent(referer) and using php (or any other server-side script) to make second request with all needed headers set.

• ^«For security reasons, these steps should be terminated if the header argument case-insensitively matches...
  ...
  Referer
  ...^»
  
  Make sure your header arguments meet the criteria from the W3C including the exact spelling.
   
• There's an annoying feature discrepancy between their Wiki and the Greasemonkey ^(GM) 0.7.20070607.0 add-on as to what is "required" to be passed using  GM_xmlhttpRequest() . Make sure you are declaring the  data  argument (even if it is set to  null  )... because in line 76 of GM's xmlhttprequester.js, it appears that GM is doing the open and send automatically. The  req.send()  call is sending the  details.data  regardless, and if it doesn't exist, the remainder of the code execution instance of GM may not happen.

Thanks for the info- I'll give the Referer bit a try shortly, im going to be frustrated if that's the issue! I'll follow up if it works.

I've also been poking around on the user groups and getting information saying it may be security related that some of the attribs are left out by the GM wrapper.

• Tamper Data add-on reports raw data and ignores other extensions (that's a big no no by the way).
   
• Live HTTP Headers add-on correctly reports the headers in cooperation with other add-ons.
   
• Firefox 2.X also caches header requests... so they will also only show up once in the logs.
   
• Greasemonkey is "sorta" responsible for not allowing the Referer header atom value to be set, but then again, sorta not!
   
• Any call to XMLHttpRequest in the chrome namespace (Firefox add-ons use this for those that don't know this) is considered "unprivileged" to modify the Referer header atom value UNLESS there is an event listener that modifies the raw data via the http channel OR there is a component add-on (I think that's XBL, still a bit noob to some of this) that is added to an existing add-on.
  
  Firefox currently has two sections, that I can find, in their trunk code (and existing releases too) that PURPOSELY BLOCKS OR CLEARS the Referer header atom value from being set in the chrome namespace.
   
• Any XMLHttpRequest call made via unsafeWindow (e.g. the native DOM object created from a web server) is considered to be "privileged" enough to set the Referer header atom value.
   

• ^«6. For security reasons, these steps should be terminated if the header argument case-insensitively matches one of the following headers:
  
  • Accept-Charset
  • Accept-Encoding
  • Connection
  • Content-Length
  • Content-Transfer-Encoding
  • Date
  • Expect
  • Host
  • Keep-Alive
  • Referer
  • TE
  • Trailer
  • Transfer-Encoding
  • Upgrade
  • Via
  ^»

Wow. Very nice Marti.. so this is starting to kind of look like an issue with FF?

Are you saying that even using XMLHttpRequest the Referer isnt being passed through FF? In other words it's currently not possible in GM to pass Referer properly?

Thanks for the (very) informative update!

Could you maybe insert a new XMLHttpRequest object in a script element into the page? That way you would have access to the Referrer.

e.g.

var xScript = document.createElement('script');
xScript.type = 'text/javascrpt';
xScript.innerHTML = "function getXmlHttpRequest(){var httpRequest=null;httpRequest=new XMLHttpRequest();return httpRequest;}";

document.getElementsByTagName('head')[0].appendChild(xScript);

var xhr = unsafeWindow.getXmlHttpRequest();
xhr.onreadystatechange = function() { 

 if (request.readyState == 4 && request.status == 200) {

  alert(xhr.getResponseHeader( 'Referrer' )); 

 }

};
xhr.open("GET", url, true);
xhr.send(null);

The header name is 'Referer', and it's a Request Header.
But according to what Marti wrote, you can use setRequestHeader("Referer", "URL") in that function and it should work.

Now that I've had enough time to cite references and document Greasemonkey a bit better (which is is important to the explanations I have here), I can get back for a while to u.s.o.

@gimmic
No prob and thanx... yes it is looking more like Mozilla is the culprit, but for reasons unknown. I still don't know whether to file a bug report with them or not... I'll search through their forums and docs soon to see if they have an official standing.

What I didn't say in my response above is about YOUR NAMESPACE... any user script namespace is in what they call in the Sandbox, and of course it is not privileged enough either to send the Referer atom (at least on the machines I have tested). Currently the only way that I have successfully sent a Referer atom is to use the unsafeWindow object, which I'm not thrilled with... I usually block all Referer atoms period, but I know some sites are annoying enough to require them. In that case I usually forge them and apparently that's not possible in chrome or the sandbox using the XMLHttpRequest (XHR for short) method, which presents a very large problem.

@Yansky
That's more or less the solution that I've come to as well, but with Descriptors comment about the extra r shouldn't be in there and depending on what method of the native DOM XHR you want to call will determine which method you use. I make typos all the time, so no big deal... but I'm doing my best to be as accurate as I can here. Unfortunately a host site will see what you do when you utilize unsafeWindow *le sigh*

@Descriptor
Thankx for catching that... I've been WAY busy with GM docs

@gimmic

I've confirmed that it's definitely the browser that is filtering it... however it doesn't look like the Greasemonkey team is going to make any effort to implement the fix anytime in the near future. Only suggestion for now is to use the DOM XMLHttpRequest in an unsafeWindow over the Greasemonkey API implementation or some combination of both to achieve what you need.