Script Reviews

Posted by Jesse Andrews on Jan 23, 2008

The last straw happened this evening. Reviews are coming this weekend! Help plan them here.

The basics: unreviewed scripts CAN be installed by logged in users only... On the rails side I'll catch the request and send a 403 FORBIDDEN if the user isn't logged in.

Why? Because people are uploading scripts *cough* scrap all *cough* whose only purpose on the site appears to be a hosting service for them. They then use the userscripts as a component of a bookmarklet.

So - what does this mean?

Every script needs reviewed before it is public. If you are logged in you can see/install everything...

So, why has it taken this long? Well, the question I have is: Who determines if a script is acceptable? I neither can nor want to review hundreds of scripts a week. Obviously when a script is updated the changes will need to be accepted before they are public.

Are the reviewers a selection of a bunch of trustworthy users?

Do we make it so anyone can, but certain users have more weight (eg, the more karma you have the more your vote counts)? And add a delay between upload and acceptance if only "weak" users have rated it... giving time (a day?) for other users to see it and vote to reject?

UPDATE

I wrote this in utter frustration. After you comments and emails we implemented hiding of scripts with "dirty words" - see Hotlinking of Unlisted Scripts.

You could comment on this post if you were logged in.

Jan 23, 2008
rahul286 Scriptwright

Nice move...
But but you should take care of a new spam which may result from this review system!
Bunch of spammers may collaborate in something like REVIEW EXCHANGE programs!

 
Jan 23, 2008
davey User

user karma is a good idea, though i wonder if you could include a weighted system so that certain keywords (for instance, scrap all) are automatically given -X and would resist simple attempts a rigging the system. or instead of keywords, maybe parse the uploaded script itself and tag it as a 'scrap all' by noting a common 'signature'.

i would also introduce a posting delay for new users - ie: newly registered users (maybe a two day probation?) cannot upload more than 1 script per day, or every hour. this will reduce the spam quite a bit too.

 
Jan 23, 2008
nungryscpro Scriptwright

Wouldn't this be pointless? As long as someone creates an account here, they'll still be able to use the scrap all scripts as bookmarklets. I think blocking hotlinking would be the most effective.

I'm always logged in so will I still be bombarded with all of those scrap all scripts since I can see/install everything.

I also see a problem with those same script creators adding in something extra that would automatically review/approve any scrap all script.

If someone updates a script due to a bug fix, will it take 3 days for the updated script to be approved. Will you keep two copies of the script on the site during the review process? The original still being public while the updated version is being reviewed.

 
Jan 23, 2008
Mindeye Scriptwright

I agree with nungryscpro. The scrap all problem seems to be the classic hotlinking problem.

How about blocking request to *.user.js files based on the referer? If those spammers use USO to host the scripts for bookmarklets in orkut, their requests won't have a USO referer. The server could deny the request, redirect them to the script's page or even serve a simple script with an alert explaining that this use of USO is not allowed

There are ways to spoof the referer, but it would help with the average user

 
Jan 23, 2008
Jesse Andrews Admin

Hotlinking wouldn't be an issue for scripts that haven't been accepted since:

12345.user.js isn't a file on the filesystem, it is a route to a rails call. In the rails call I can see if the user is logged in or not, if they aren't they are sent a 403 FORBIDDEN.

Scripts that are accepted could be "hotlinked".

The reason I don't want to disable that is that there are valid usecases of grabbing the .user.js without visiting us.o first, and you cannot trust referrer (some firewalls remove it, and there are techniques for removing it via nested iframes)

 
Jan 23, 2008
Mindeye Scriptwright

But, if I understand this system, a bookmarklet spammer could get the file even if it hasn't been approved if he/she is logged in. I fail to see how this will address the "scrap all" scripts flood. And, as nungryscpro said, a legitimate user who is always logged in will still have to bear those scripts

 
Jan 23, 2008
Britt Selvit... Admin

This is something we've been thinking about for some time.

One idea for a Review Board system is this:

1) The number of reviewers starts relatively small. Probably around 10 users or so.

2) User X gains karma by a number of means. Highly ranked form posts, excellent scripts, insightful comments, etc.

3) At some point user X is flagged as "Potential for Review Board," in which case USER X is reviewed by someone already on the board.

4) He get's accepted or denied, with the potential to be reviewed again.

5) Problem users can be handled in traditional moderation style fashion.

Regarding logged in users being able to see the scripts still ... we of course already have the ability for a script to be marked as totally malicious and spam and hidden on the site, and a Reviewer could mark scripts as such.

This also segments us into the extremely nice position of being able to offer scripts with "Stamps of Approval" that are known to be not only well written, but extremely useful for users.

 
Jan 24, 2008
KARASZI István Scriptwright

and what then if you need to pass a CAPTCHA before install

and in this case hotlinking could be fixed, you don't need the referer check and so on.

 
Jan 24, 2008
SlimShady User

Britt, that sounds like a great idea!

 
Jan 25, 2008
Justin Ormont Scriptwright

Perhaps obvious, but why can't we just simply disable hot-linking for all scripts? Why do we need to directly link to the .js? Force people to link to the /scripts/show/ instead. I suppose update scripts may want to hot-link, but tell them to get their own hosting too. My main script, gets updates from a secondary server. The upside to a secondary server, is that script updates don't count towards my install count, only initial installs.

Summary: We could always just check the HTTP_REFERER header and disallow blank entries such as bookmarklets likely have.

 
Jan 25, 2008
thorbenhauer Scriptwright

I've never paid attention to the 'scrap all' scripts, but if they are just using this site as a hosting service, I'll second the comment by KARASZI István.
By solving a captcha before installing a script no regular user is really harmed. Hotlinking by bookmarklets could be easily prevented without a complex review mechanism.
I'm using hotlinks by myself, but it would be no problem to link to the script site or a self hosted version instead.

 
Jan 25, 2008
TwistydMorticia User

I think what Britt is suggesting is spot on. I think that Stamps of Approval would help tremendously as, an end user, I do worry about using a malicious or defunct script.

I think you should pick a trustworthy bunch of reviewers that are suited to job / qualified and definitely add a delay, just my opinion.

 
Jan 25, 2008
thorbenhauer Scriptwright

Reviews, ok nice to have. But the main problem is, who does the job? Who hands out karma? On which basis? 'Highly ranked', 'excellent', 'insightful' - who decides about these classifications?
Have a look at amo. Do they _really_ check each extension and each update of an extension line by line?
Ok, I don't want to start a discussion about the last issue. The main point is: if you could beat them with technical limitations that don't need a hell lot of work to maintain, why not?

 
Jan 25, 2008
nungryscpro Scriptwright

I like reviews and I'd love to see them imlpemented... but using it as means to stop scrap all scripts doesn't seem like the right way to take care of the problem. Using captcha sounds like a great option if it's feasible

Regarding reviews if it's just a select few that do the reviews then what if they don't use the site the script is meant for. I've already seen that a couple orkut scripts were blocked that weren't scrap all scripts. I guess that's where it might fall into the communities idea.

On the otherhand, if reviews are left up to the public (even just members) then we'll see biased reviews just like there are inflated install counts. We'll need reviewers to review the quality of the reviews.

 
Jan 26, 2008
BlindWanderer Scriptwright

I don't write popular scripts. I write little scripts that do little things. I release my scripts to the public for the unlikely case that there is someone else out there who wants to do the same things. A review system would make it harder for people to find and install my scripts because most of my scripts wouldn't be reviewed, there just isn't enough interest in them.

I think scripts should be assigned a score based on review, the score starting at zero. Any script with a score of zero or above should be allowed to be publicly installed/viewed. Scripts with negative score would disable hotlinking. Score would not start to be calculated until two reviews. A review could positively or negatively effect the score.

In addition scripts could be placed into 3 categories based on score: Positive, Negative and None; None would be for scripts that didn't have 2 reviews.

 
Jan 26, 2008
jerone Scriptwright

My suggestion:

Add a CAPTCHA against bots and bored users.
Then assign like 10 moderators (more after a while) who just random clean up the scripts: spam and duplicated.
Not only scripts but also for the forum. It has way to much topics in too little threats.

Don't use a review system; to hard to check.
Don't hide some scripts; gives only more problems with the makers.
Don't assign names to 'potential' scripts; just allow it or delete it.
Don't work with ranks; doesn't say anything. Ranks should be just the mods.

 
Jan 26, 2008
jesses Scriptwright

The result of manually reviewing each script will be less spam scripts being displayed but it's unlikely to have much effect on the number of spam scripts being uploaded, thus the amount of work to keep the site spam free will remain high.

As mentioned earlier ff you also block hotlinking and redirect people to a message explaining the situation it should help to educate the orkut users and hopefully reduce the amount of spam scripts that are being uploaded.

 
Jan 27, 2008
Descriptor Scriptwright

Hotlinking - don't block empty referer, just block specific sites.

 
Jan 27, 2008
jk- Scriptwright

Any community-oriented review system will only be abused in my opinion.

The answer seems simple:
1. Block multiple scripts with the same @name.
2. Force userscripts to have @include set

If you create a system where a user can have "more weight", those with the fake accounts will simply boost the point total of whichever fake accounts they wish to make into "heavyweights".

Every site that institutes a "karma" system, (or a "you'll get a cookie if you wag your tail when I say so" system) has these problems and quite frankly, it's the type of thing that these brainless kids on orkut would probably want.

 
Jan 30, 2008
Jesse Andrews Admin

jk- I understand your concerns and I have the same! That is why we've not implemented it yet - I want to make sure it isn't breakable.

 
Feb 20, 2008
ricky2008 User

How to make a script Im new here so I dont know that

 
May 6, 2008
tere User

its very nice move ,good idea

 
Jun 11, 2008
merttt User

Thank you very much! i like this blog, too.

 
Jun 29, 2008
TylerRus User

Just test of my gravatar...

 
Jun 29, 2008
TylerRus User

Bad... It seems to be waiting