Superior Searching

Posted by Jesse Andrews on Oct 20, 2007

Searching for userscripts used to be slow and only search the descriptions. Now searching uses SOLR and searches names, summary, descriptions and source.

Searching on this site always sucked, for you and me... Every day I get a report of the the performance of the site (rails performance analyzer). I always see:

Slowest Request Times:
	ScriptsController#search took 69.622s
	ScriptsController#search took 64.092s
	ScriptsController#search took 55.131s
	ScriptsController#search took 49.324s
	ScriptsController#search took 48.399s
	ScriptsController#search took 47.859s
	ScriptsController#search took 45.436s
	ScriptsController#search took 44.657s
	ScriptsController#search took 43.686s
	ScriptsController#search took 42.745s

70 seconds for a search!!! That is bad for everyone - who wants to wait for over a minute for results, and I have to make it scale. Luckily the average search time is less than a second (here are the stats for the last day):

                                         	Count	Avg	Std Dev	Min	Max
ScriptsController#search:                   	12366	0.728	2.256	0.128	69.622

I had been hoping Henrik would have time to add acts_as_solr to the site since he is doing lots of cool stuff with it on other rails projects, but alas, he is busy. So I rolled up my sleeves, and released solr based search.

The search order is currently only ranked on the prevalence of the search in the document (ignoring number of favoritings, installs, comments, ...) but it is a start. Once this is stabilized adding better sorting shouldn't be too hard.

Now if only the front page didn't suck so bad... ;)

You could comment on this post if you were logged in.

Oct 21, 2007 LouCypher Scriptwright	Good work! Thanks, Jesse.

Oct 21, 2007 Raffles Scriptwright	Great news. Searching before did indeed suck! Off to try it out now...

Oct 21, 2007 David Hunter User	Cool, thanks. That would be great if it gave the option to sort by the other criteria you mentioned as well!

Oct 21, 2007 cyberPIN8 User	The problem now is the way the results are sorted out. It was better before when they where ordered from most recent to oldest updated script. Now I see scripts from 2006 next to the ones updated only hours ago. Hope you guys fix this.

Oct 22, 2007 snooze User	"The problem now is the way the results are sorted out. It was better before when they where ordered from most recent to oldest updated script." I think so ! impossible to see if there's new or update script !

Oct 22, 2007 Jesse Andrews Admin	I've been reading on sorting options - right now it is sorting by "score" - I'll add in sorting by create / updated date

Oct 22, 2007 Jesse Andrews Admin	Search results are now ordered by the most recent updated source first

Nov 4, 2007 NukePrice.com Scriptwright	Wonderful!! Thanks!

Nov 27, 2007 Wellsie1116 User	hmmmm, spam

May 6, 2008 ersin User	thank you. very good news. `ersin`

Jul 15, 2008 merttt User	.Cool, thanks. Wonderful!! Thanks!.