Discussion on Unpaginate pagination microformated web pages | Userscripts.org

Quick question, replied by ak2344

ak2344 — Mon, 16 Nov 2009 17:07:35 +0000

Hello,

I'm not the best w/ Jscript and Regex. Do you know what line/ script I would use to apply the unpagination to the "Matches" page on OKCupid? It would be a huge help!

Thanks.

Need Help using your script, replied by StevenD

StevenD — Sun, 19 Apr 2009 00:23:48 +0000

Thank you very much for your reply! I will certainly take a look at the scripts you listed (I didn't know we could search by tags here!)
I am going to try what you have suggested and let you know! Thanks again for your reply.

Need Help using your script, replied by Johan Sundström

Johan Sundström — Sat, 18 Apr 2009 19:03:01 +0000

Potential candidates for unpagination is any page that was cookie-cutter-template-wise turned into multiple pages between which there are "next" links, or "page 1..N" links, of any sort. Anything you can craft an XPath expression for picking up on -- //a[.="next"] is an easy case, but it can be anything that is deterministic enough to be expressed in the language, which is very rich in its possibilities (peek at the very minimal "source code" of some of these scripts for inspiration, if you are new to it).

When I look for XPath expressions, I use Firebug's Inspect feature a lot -- open the Firebug console, click "Inspect" on top, and click on the next link. Let's say we're on page 2 right now and it comes up with an HTML structure that looks like this (a somewhat common and maybe-not-that-trivial one, for a budding XPath hacker):

<div id="comments">
  <a href="...">1</a> 2 <a href="...">3</a>
</div>

My strategy in this case will be "among all the page links, select the first one that follows a not-just-whitespace text node", which will probably work on page 1 to N-1, which happens to be all pages we need a next link from. (For a tricky case where multiple strategies are needed, craft an expression like 'how-to-do-it-on-some-pages | how-to-do-it-on-other-pages | how-to-do-it-on-yet-other-pages | ...'.)

In this case, the above comes out, for instance (untested -- if you have some example page in mind though, feel free to name its URL), id("comments")/a[(preceding-sibling::*[1])[length(normalize-space(.))>0]]

It's easy to try the expression in Firebug; just try $x('id("comments")...') in the Firebug console and see what it found. If the first node is your next link, and if that is the case on not just this page, but any page in the sequence (try the first page, one in the middle somewhere, the last but one page and the last one -- which should come up empty), and you'll probably be reasonably certain you got it right.

Facebook evolves quicker than the frequency of my visits or active use of the site; my hack has in all likelihood rotted long ago.

Need Help using your script, replied by StevenD

StevenD — Sat, 18 Apr 2009 04:10:18 +0000

Hello Mr. Sundstrom.

First, thank you for sharing this script and also for your blog. It is very thought provoking!

I have a couple of questions for you, if you don't mind. How do I identify potential candidates for your script? (and) How do I then go about writing the customized calls to the unpaginate function?

For my first question, does this script only work with sites that use 'NEXT'? I am interested in unpaginating a site that includes links to the numbered result pages, but does not provide a next link. If this script would work with numbered links that lack a 'NEXT' link, what would my custom script look like (any examples?)

Secondly, I looked at the facebook search unpagination script you have but was unable to match up the xpath statements with current search results. Does that script still work? If yes, then may I ask how you derive the id's you have in the script. I ask that since I would like to create a customized version for the site I visit.

I hope my questions are clear and I appreciate any time you provide in answering my questions! Thanks in advance, StevenD

New feature: craft unpaginators helper, replied by Johan Sundström

Johan Sundström — Thu, 04 Dec 2008 12:21:43 +0000

If you run the bookmarklet javascript:void(craft()) on a page you want to unpaginate, you get a crude:ish tool at the bottom of the page where you can edit in the two or three xpath expressions you tend to need to make a new unpaginator: the xpath to items that should be pulled into the page on the far left, next the xpath expression that finds the "next" link on the page, and finally (optional) an expression that finds the pagination bar, if there is one. The second xpath expression may now be specified as a relative path from the panel, if there was a panel.

(If a panel is specified, on unpaginating, that panel will be replaced wholesale with the contents of the next page, so it, eventually, as all pages have been unfolded, lists what the final page had in that section.)

I'm sure this is way too technical and ill described for most people that are not myself, but it's at least something, and it really only caters my own creature comforts at the moment, so I didn't exactly go overboard on the user interface parts. :-)

Archived Comments, replied by Johan Sundström

Johan Sundström — Wed, 17 Sep 2008 00:29:21 +0000

In part for not knowing about the other one (my search skills also wouldn't have figured out that Autopagerize removed rather than added paging, though) but first and foremost, this was an experimentative study I had been brewing on for a while, in writing generic cooperating user scripts that meet one another over a micro format (stigmergy), augmenting the microformat in different ways.

This key bindings script augments the same microformat as the unpaginator, but in a different way: instead of pulling in next page's items into this page, it adds keyboard bindings to scroll to the next item on the current page, so you can move up and down the item list by pressing m / p (rather arbitrarily picked, to sit well with how I read online comics, which uses j, k, l and n for related purposes implemented by same script and another), over and over again.

As you'll note, the two scripts run independent of one another, so if you install both, you get both features -- infinite scrolling and keyboard navigation of the items, even to new items that get added as you tap your way through them.

The main beauty of all of it is that all three (or more) parts can be maintained, developed, improved, added or removed independently of one another: any scriptwright can add a new site to the system, without communicating with me or anyone else, anyone that has a use for the properties these microformat consumers use can make new scripts using them for whichever purposes, and you don't have to opt in to all features making use of the data, only those you like.

(Based on the data items Autopageresize adds, it does not do precision cutting of only the item nodes from the pages it unpaginates, as my variant does, but if all one wants is something to undo pagination that system might be the better pick, given its large user base.)

Archived Comments, replied by Paul Irish

Paul Irish — Mon, 08 Sep 2008 17:51:58 +0000

Johan,

Been a fan of your scripts and blog for a long while..
But I must ask: Why reinvent this wheel?

Autopagerize (http://userscripts.org/scripts/show/8551) has been doing this for a long time, and has a great collaborative filtering setup already defined.. (And arguably its a cleaner and more efficient system than individual scripts for each site)

Is there a reason Unpaginate was made a separate effort?

Archived Comments, replied by Johan Sundström

Johan Sundström — Wed, 25 Jun 2008 22:16:33 +0000

Fun, isn't it? :-)

Archived Comments, replied by gollum

gollum — Mon, 23 Jun 2008 11:03:59 +0000

Also - couldn't help myself - one for 'userscripts.org/scripts' http://userscripts.org/scripts/show/28968

Archived Comments, replied by gollum

gollum — Fri, 13 Jun 2008 07:22:12 +0000

Hi Johan, have created a producer for phpbbV2 sites - at least it works on http://www.ibdof.com/viewtopic.php?t=650

Unpaginate phpbbV2 threads

Archived Comments, replied by Johan Sundström

Johan Sundström — Fri, 16 May 2008 09:01:15 +0000

Aquilax: if you haven't already, feel free to copy and edit this script to do what you like. You won't have to change the microformat producers, and if you want your custom behavious only on a few sites, you can regulate that with the @include and @exclude patterns. Good luck!

Archived Comments, replied by Aquilax

Aquilax — Sat, 22 Mar 2008 16:53:57 +0000

Hi Sundström, I was thinking that it would be nice to have an "event" to change each result in the page. What I'm trying to say is that it would be useful to have a way to register a function that will be called for all the original results in the page and for all the new added results.
I'm using the Pagerization script and I find very nice how it change the results on the google.com/images.
I'm using your script to unpaginate some photo albums and with such a function I would be able to add a mouseover event and show the picture in a float div and not have to load them each time in separate page to look at them.

Archived Comments, replied by Aquilax

Aquilax — Thu, 28 Feb 2008 18:54:09 +0000

OK, it was my fault, I removed the "pagination-container" and now it works fine. Great work. But I think that it has some problem to detect the scroll to the end of the page, sometimes I have to scroll twice o three time before it load the next page.

Archived Comments, replied by Aquilax

Aquilax — Thu, 28 Feb 2008 17:55:40 +0000

Ok I've "cloned" the "OkCupid journals" scripts, it works, but instead of append the next page it replace the actual page. The layout of the page is a table with 4 rows and each rows has 4 cells, for a total of 16 pictures per page.
I set the "pagination-container" to the table body of the table and the "items-xpath" to the table rows.
The results is that each 4 seconds it loads the next page and replaces the content of the table body with that of the next page instead of adding it at the bottom of it.

Archived Comments, replied by Aquilax

Aquilax — Thu, 28 Feb 2008 17:39:27 +0000

Uhm I'm danger adverse, some times, I have a lot of micro scripts that I have done only for myself and I don't think that they would be interesting for any one else so I never uploaded them on userscripts.org. One of them would have used your microformated script, I wanted to add the "endless page" effect to a pictures collection page. I tried to copy your yahoo script but it didn't work, which is the only one script that works with GM 0.7? So I can copy that script and try to adapt it.

Archived Comments, replied by Johan Sundström

Johan Sundström — Thu, 28 Feb 2008 16:04:18 +0000

My remotely hosted version (small link below the install link) is 0.8 only, though, but sports a live editor for microformat producers, for advanced xpath wranglers, which you activate by running the bookmarklet javascript:void(craft()) (be sure to be logged in on userscripts.org too, or I assume their token verification for uploading scripts will fail).

(You also might want to change namespace from what I put in mine before you post.)

Archived Comments, replied by Johan Sundström

Johan Sundström — Thu, 28 Feb 2008 15:58:53 +0000

The version installable here works without 0.8, but all but one of its microformat producers (that I've made, that is) don't.

<Beware!>

The development build I run at the moment is here, but it is potentially dangerous and untested in the large; it may quite possibly wipe your entire list of GM scripts, so be sure to backup if you consider running it (we have fixed/regressed that symptom three times already, so it could be there still, or again).

</Beware!>

(Also, I could be a shady guy linking a malware infested xpi; there are lots of good reasons to wait for a real 0.8 once available.)

Archived Comments, replied by Aquilax

Aquilax — Thu, 28 Feb 2008 15:35:06 +0000

Interesting but without greasemnkey 0.8 doesn't work. Where is possible to download and install the pre-release version?

Archived Comments, replied by Johan Sundström

Johan Sundström — Mon, 25 Feb 2008 03:03:46 +0000

I wrote up a blog post about the technical stigmergy / ecology / microformat aspects of this script and its many companions that is probably of interest to fellow scriptwrights wanting to extend this hack to other sites or consume the same data it does. Click the "pagination microformat" tag on the right to find them all.

Archived Comments, replied by Jesse Andrews

Jesse Andrews — Wed, 17 Nov 2004 01:05:46 +0000

The following is an archive of comments made before threaded discussions was implemented (November 16th, 2008)