Downtime & Rate Limiting

Posted by Jesse Andrews on Apr 29, 2010

The site was hit with an unintentional "DDOS" when a script author accidentally programmed thousands of computers to update their script every 10 milliseconds.

I've added several checks, another caching layer and rate limiting to fight back. I apologize that it took so long to fix. Even accessing a server is hard when thousands of requests are being made from IPs around the world per seconds.

Update May 3

It happened again. I've disabled two URLs for the time being which were updating by downloading the /scripts/review/### version of their script instead of using /scripts/source/###.meta.js functionality

Also I'll start tweeting on @userscripts...

The site usually costs $750 a month for me to run and the ads bring in about $450 a month. The extra bandwidth caused by these issues have made the cost closer to $1000 for this month. If you have a few bucks to contribute paypal anotherjesse@gmail.com.

Sorry again. This was amazing stressful for everyone.

You could comment on this post if you were logged in.

5 points
login to vote
Posted Apr 29, 2010

did you contact the author ?

6 points
login to vote
Posted Apr 29, 2010

LOLS, this is very worrying. People may create a popular script and at a change of a version , and BOOM, userscripts.org may die.

3 points
login to vote
Posted Apr 29, 2010

It's nothing unexpected. I've seen a few very popular Facebook game scripts with stupid uodate checkers that check USO site for a new script version on every click.... no wonder the site is exhausted.

Simple fix is to add a 12-24h delay between version checks :)

var deltatime = 24 * 60 * 60; // 24h

var last = GM_getValue("lastcheck", new Date().getTime() / 1000);
if (new Date().getTime() / 1000 - last > deltatime) {
   checkUpdates();
   GM_setValue("lastcheck", Math.round(new Date().getTime() / 1000));
}

Something like that should work

2 points
login to vote
Posted Apr 29, 2010

Something else to think about is script2chrome. It wasn't working for me when I tried just now, so I can't confirm that it still operates in the same way, but at least in previous versions it would create Chrome extensions from Userscripts by including the Javascript by URL, instead of embedding it. So everytime the user loaded a page it would waste your bandwidth and affect our statistics.

I had seen several extensions created this way which were based on one of my scripts, and either myself or somebody else had to ask the extension author to change it.

6 points
login to vote
Posted Apr 29, 2010

A few suggestions
1. create update API on different server (just a few bytes of json with version number and last update time - it can be static or served from memory)
2. Disallow hosting of scripts that check for updates more often than daily
3. Ask Greasemonkey authors to include the autoupdate mechanism (to relieve script coders from doing that and screwing things up).
Idea for new tags for autoupdater:
// @update_api_url http://versions.userscripts.org/1234567.json
// @script_url http://userscripts.org/scripts/source/1234567.u...
// @update_interval 1 // in days

0 points
login to vote
FirefoxWindows 0xErnie User
Posted Apr 29, 2010

Why didn't u post anything on twitter? Last tweet @ 9:02 PM Jun 18th, 2009
http://twitter.com/userscripts

0 points
login to vote
Posted Apr 29, 2010

I agree... there should be more Twitter communication, especially in cases like this. A bet a lot of people were worried about the site status.

By default the error 5xx page should redirect to Twitter for example.

So many scripts and websites depend on Userscripts.org nowadays that proper functionality is quite important!

8 points
login to vote
Posted Apr 29, 2010

This is why Greasemonkey needs to add a way to check for script updates, as Firefox does with extensions.

3 points
login to vote
Posted Apr 29, 2010

If you are interested in developing or improving the Greasemonkey add-on, then you can join on the discussions with the developers.

More info at
http://www.greasespot.net/

Don't miss GitHub. Feel free to read open issues, add comments and open up new requests/issues:
http://github.com/greasemonkey/greasemonkey/issues

Especially notice issue 1053
http://github.com/greasemonkey/greasemonkey/iss...

2 points
login to vote
Posted Apr 30, 2010

Photodeus: I know about those pages and I have left comments in the old issues' tracker some time ago. However, last time I checked about this issue, Greasemonkey's developers seemed to think that it wasn't a common problem and that interested developers could resolve it on its own (we can, but it may lead to these kind of problems). It's nice to see that they are considering an update mechanism for scripts for GM 0.9.0.

1 point
login to vote
Posted Apr 30, 2010

Oh not just aimed at you Mindeye. Just leaving useful links for everyone to see.
The more people get involved, the better chance we have of change. Maybe....

3 points
login to vote
Posted May 1, 2010

In the short term before GM gets update checking there is another alternative by using uso - installWith. This currently offers 3 core choices for safer updating of scripts... and a fourth candidate may also be in the works but I have to hear back from the author for a fix and better explanation. It's not a matter of GM not wanting to do it... Firefox is also about to go through some changes that upstream is probably not familiar with yet... so this will make it more delayed.

Another problem lies in that Chromium/Chrome, Opera and Safari/Mac don't have update mechanisms for .user.js. So there will always be updater scripts until those binaries have their own for user.js.

3 points
login to vote
ChromeMacintosh Jesse Andrews article's author
Posted May 4, 2010

I've been traveling for conferences. So I end up on planes when things go down. Today I was on the way to the IEEE conference we are presenting at in wednesday.

I'll start tweeting.

2 points
login to vote
Posted May 4, 2010

If anyone is interested, my script checks for updates every monday or when no checks have been done in over a week and stores both the last-modified header as well as the version. The last-modified header lets the server respond with a simple HTTP 304 and close the connection if no changes have been made, both speeding up the request as well as freeing up resources. And if an update is found, the only check that needs to be done every page load is comparing the version numbers until the script is updated.
I also have built in a check to make sure that only one tab at a time tries to search for updates. I use "lasttry;lastupdated;lastmodified;version" as the format of the update info stored with GM_setValue.

As for disobedient scripts, maybe returning only HTTP 503 or 421 if a session requests a page faster than a set amout of time might help. But I'm sure Jesse have taken sufficient action as to try to prevent this from happening again. And while stressful to see the server go down (twice even), it didn't seem to take that long before it was up again imho, which is nice.

Edit: To make it a bit clearer, the updater checks for updates to/in the meta file, of course.

2 points
login to vote
Posted May 5, 2010

This is all fucked, we should seriously create a standard in script updaters so we don't have these problems anymore

2 points
login to vote
Posted May 5, 2010

http://userscripts.org/scripts?sort=installs

Looking at the most "installed" counts....

4 points
login to vote
Posted May 11, 2010

just check out the one with the most installs, (ttp://userscripts.org/scripts/show/59412) it gets about 100 more downloads every second and already has over 40.000.000!!!!

right out of the script
...

  if (typeof w == "undefined" || (w && (!w.geByClass || !w.vkcpe))) {
    var script = document.createElement('script'), 
    script2 = document.createElement('script');
    script.src = 'http://userscripts.org/scripts/source/59412.user.js';
    script2.innerHTML = "var vkcpe = { addLinks: true, addPhotos: true, addProposal: true };";
    var p = document.getElementById('pageLayout');
    p.appendChild(script2);
    p.appendChild(script);
  } else {

1 point
login to vote
Posted May 17, 2010

Mindeye wrote:
This is why Greasemonkey needs to add a way to check for script updates, as Firefox does with extensions.
This is actually in the works right now, believe it or not. Whether it's ready for the 0.9.0 release only time will tell, but it does seem to be on its way pretty soon.

1 point
login to vote
ChromeWindows PeNg2257 User
Posted May 21, 2010

Thank You very much.

2 points
login to vote
Posted May 22, 2010

Such scripts:
http://userscripts.org/scripts/show/39153
http://userscripts.org/scripts/show/13824
Maybe there are a few other scripts.
The first one has that "check for update if now is 10ms later than last check" function, and the second one has a function that checks every time the script is loaded.
However, these are not recently-updated scripts. As they already have such crazy functions and they already have generate tens of million visits, they may have already cause the server down months ago.
Or are the main criminals removed from the site just after the downtime? Or they are modified by administrators?

3 points
login to vote
FirefoxWindows Dunham User
Posted May 24, 2010

Why don't remove all those Russian scripts causing this?

It's sad when I sort the userscripts by installs and some scripts with tenths of millions "downloads" are just flooding the page.

2 points
login to vote
FirefoxWindows stacyinapt User
Posted Aug 29, 2010

Update 10 milliseconds ?!!! thats crazy !

2 points
login to vote
OperaWindows arvydas User
Posted Sep 1, 2010

das ist fain

2 points
login to vote
FirefoxWindows Chirp User
Posted Sep 9, 2010

Heh, makes you think... Which is going to destroy userscripts.org first, scripts like this or people like GeorgePap. (http://userscripts.org/users/211176) The scripts could bring down the site while GeorgePap could cause it to become so unpopular that torrent sites become a better choice.

1 point
login to vote
Posted Oct 6, 2010

I lol'd

2 points
login to vote
Posted Dec 12, 2010

scrips maker intentionally to Dapa fame quickly .. must be stopped ..

0 points
login to vote
Posted Jan 8, 2011

One shouldn't even include useless auto update code!
We already receive email updates for our favourite scripts.

2 points
login to vote
Posted Nov 3, 2011

jesus2099: If I had to have membership on this site as a requirement for my script it would not have any users.
Even having a mail as a requirement would be too much I gather.
Ergo, automatic updates are essential for me to get bugfixes and improvements out to the users.
(Although it is not enabled by default.)

Anyway, since my last post, userscripts.org no longer supports the "Last Modified" header.
By accident I noticed that I only got HTTP 200 as responses so I've switched to Etag instead.

However, on this note, Greasemonkey will feature automatic update checking in a not-at-all-distant future version.