Dilettante's Ball

Category: coding

  • On unAPI

    After arguing with Dan for hours on how to implement unAPI, I decided to take him up on his request and implement an unAPI service for our OPAC.  This way I would have some real world experience to back up my bitching.

    So, here you go.  This script takes our SRU to OpenSearch services for the OPAC and the Universal Catalog.  I’ve modified the OpenSearch CGI to include the unAPI identifier.

    The unAPI service is written in PHP and calls the SRU service and presents the results accordingly.

    So, my reactions:

    1. The JSON requirement doesn’t really bother me.  Whatever is chosen is rather arbitrary, so I’m ok with JSON.  I’d be just as ok with XML or text delimited.  I see no real difference.
    2. I am not entirely convinced that just exposing the metadata (with no wrapper) is scaleable.  I’m not saying I can’t be convinced, I’m just saying I’m not currently.
    3. I see no point in sending the status along with the response.
    4. This is, indeed, much simpler than making an OAI-PMH server in PHP over Voyager.
    5. I would really prefer parameters to paths.  I think there are too many assumptions about the implementor (and the future) to use paths.

    What I am still not entirely sure about fundamentally, however, is why we need another sharing protocol.  Dan claims it’s because OAI-PMH and ATOM are still too complicated for stupid people and the less technically inclined to pick up and he wants something simpler.

    What I don’t understand is why we don’t just make a “simpler” API to one of those protocols.  Choose a particular syndication protocol (after all, that’s really what OAI-PMH is) and then just make an API to “Gather/Create/Share” with it.

    Personally, I am much more interested in making our OAI-PMH archives and our SRW/U servers available via ATOM, much like we made SRU available to OpenSearch.  That way we pick up on the wide variety of tools already out there.

    This interests me on other levels, since I’m really starting to picture the Communicat as a sort of ATOM store.

    I see a lot of potential with unAPI (a whole hell of a lot of potential), but I would rather utilize an existing protocol (especially one that promises to have a lot of users, read: ATOM) than build another library system that is largely ignored outside our community.

    On a related note, I want to point out what I see as two wastes of library developers’ time:

    • Walt is right:  DON’T MAKE LIBRARY TOOLBARS (it’s in there, trust me).  It’s asking too much to expect people to install and use something that has such a limited scope.
    • The OPAC jabberbot follows the same line of thinking.  In #code4lib, our resident bot (Panizzi) employs an OpenSearch client.  Honestly, this makes tons more sense since it’s easy (honest!) to make your OPAC speak OpenSearch and there are going to be a lot more useful things available via OpenSearch than a handful of library catalogs.

    No, I think we’re better served making our data work contextually in a user’s information sphere.  Push our data out to common aggregators rather than replicate the services to handle our arcane practices.

  • Suck on this, Diebold

    All in all, I’ve been pretty unhelpful in the coordination of Code4Lib 2006. Those that know me know I’m not terribly organized and horribly forgetful, so I knew it wasn’t in anybody’s best interest for me to commit to much. With that, I felt it best if I was involved with the proposal submission/vetting process.

    Early in the organizing process, the “organizing group” (committee is far too strong a term) set up a Backpack page to keep up with all of our tasks and documents. It worked pretty well. I liked the idea of a centrally located “document” that was editable by several users (the “proposal submission group” was made up of myself, Jeremy Frumkin and Dan Chudnov (on a non-committal basis)), so I used another Backpack page to compile the submissions. I am glad that I did this; I would never have been able to keep up with them, otherwise.

    As we got closer to the submission deadline, it dawned on me that I needed to set up the voting mechanism quickly (we had decided to allow any potential attendees vote on the program they wanted to see). Roy Tennant had toyed with the voting module in drupal (since that’s what code4lib.org runs), but it wasn’t quite what I had pictured.

    What I wanted was:

    • A page where the user could see all the proposals at once
    • A simple up or down vote
    • A way to tally how many “yea” votes the user has cast (since we were allowing them to cast 11 votes)
    • A way for the user to revise their votes
    • An auditing system
    • Something to tally the votes for me

    So I made my own. One nice side effect to having compiled all of the submissions in Backpack was that it was very easy to screenscrape. All of the “notes” are contained in divs that have the class “note”. The id attribute contains a unique id for that note and the titles are contained in an H3 within that div. Perfect for scraping.

    The first thing I did was set up a simple MySQL database that contained three tables: voters, proposals and votes. Voters probably wasn’t necessary, but I had this original plan to “register to vote” within code4lib.org and then the voting mechanism would “review your registration”. It wound up being much less sophisticated. So, basically “voters” was two fields, drupal user name and id.

    “Proposals” was also just two fields: id and title, and the intention was that this would get scraped from the Backpack page (the id being the unique “note” id). “Votes” was just a joining table between voters and proposals.

    Next, I made a bookmarklet that rewrote the Backpack page to have a bunch of forms with submit buttons on them within the “note” divs. There’s a lot of wasted space in Backpack, so I inserted an Iframe under the Backpack logo in the right column to tally the user’s votes. The bookmarklet was pretty simple and can be found here.

    Next, I wrote a PHP script that set in the Iframe and tallied the votes. When the user posted a vote, it would check to see if the proposal had already been added to the proposals table (if not, it would add it) and add the user’s vote to the “votes” table. It would output all that they had already voted for and let them know how many votes they had left. It would also create links next to each submitted vote for the user to remove a cast vote.

    The last step was making a node in Drupal that create made a bookmarklet link that included the user’s username (the user needed to be logged in to vote). This step, sadly, took me the longest because I had no idea how to get the username in Drupal.  It turns out you can “global” the $user variable and have access to all its attributes.
    Pretty simple.

    Lastly, I created a handful of webservices to keep a running tally of which proposals had enough votes to “win”, a list of registered voters and how many votes they had cast, and (when more reshuffling the schedule allowed 3 more presentations to be accepted) a list of all proposals and their vote counts.

    For throwing something together at the last minute, it worked ok. There is one user that doesn’t have a username (no clue who that is or why it happened) and Ed Corrado doesn’t appear in the registered voters output because his “voting cookie” was set before I cleared out all of the “test votes” (I think).

    All in all, a fun little hack.

  • Rails Resolver Router: on Rails!

    Since my foray into python a couple of months ago, I’ve been enjoying branching out into new languages.

    I had pitched the concept of a link resolver router for the state universal catalog to a committee I sit on (this group talks about SFX links in the 856 tag and whatnot). The problem with making links for a publicly available resource point to your institutional resolver is just that. It’s pointing your your institutional resolver, despite the fact that your audience could be coming from anywhere. This plays out even greater in a venue such as a universal catalog, since there’s not really a “home institution” to point a resolver link, anyway. OCLC and UKOLN both have resolver routers, and OCLC’s certainly is an option, but I don’t feel comfortable with the possibility that all of our member institutions might have to pay for the service (in the future). My other problem with OCLC’s service is that you can only belong to one institution and I have never liked that (especially as more and more institutions have link resolvers).

    So, in this committee I mentioned that it would be pretty simple to make a router, and since I was having trouble getting people to understand what exactly I was talking about, I decided to make a proof-of-concept. And, since I was making a proof-of-concept, I thought it’d be fun to try it in Ruby on Rails.

    Now, a resolver router is about the simplest concept possible. It doesn’t really do anything but take requests and pass them off to the appropriate resolver. It’s a resolver abstraction layer, if you will. I thought this was a nice, small project to try to cut my Ruby teeth on. There’s a little bit a database, a little bit of AJAX. It’s also useful, unlike making a cookbook from a tutorial or something.

    It took about three days to make this. After you pick your resolver (Choose a couple! Add your own!), you’ll be taken to a page to choose between your various communities for appropriate copy.

    I chose this particular citation because it shows the very huge limitation of link resolvers (if you choose Georgia Tech’s resolver and Emory’s resolver, for instance); despite the fact that this is freely available, it does not appear in my resolver. That’s not really the use case I envision, though. I am thinking more of a case like my co-worker, Heather, who should have access to Georgia Tech’s collection, Florida State’s resources (she’s in grad school there), Richland County Public Library (she lives in Columbia, SC), and the University of South Carolina (where her husband is a librarian). The resolver router alleviates the need to search for a given citation in the various communities (indeed to even have to think of or know where to look within those communities).

    Sometime later this winter, I’ll have an even better use case. I’ll keep that under wraps for now.

    Now, my impression of Ruby on Rails… For a project like this, it is absolutely amazing. I cannot believe I was able to learn the language from scratch and implement something that works (with this amount of functionality) in such a short amount of time. By bypassing the need to create the “framework” for the application, you can just dive into implementation.

    In fact, I think my time to implementation would have been even faster if the number of resources/tutorials out there didn’t suck out loud. Most references point to these tutorials to get you started, but they really aren’t terribly helpful. They explain nothing about why they are doing what they are doing in them. I found this blog posting to be infinitely more useful. Her blog in general is going in my aggregator, I think.

    When it comes to learning Ruby, this is a masterful work of art… but… not terribly useful if you just want to look things up. I recommend this for that.

    Anyway, I am so impressed with Ruby on Rails that I am planning on using it (currently) for “alternative opac project“, which is now being code named “Communicat”. More on this shortly (although I did actually develop the database schema today).

  • But, you see, my wheel is nothing like all of those other seemingly identical wheels

    I am still feeling my way around Python. I have yet to grasp the zen of being Pythonic, but I am at least coming to grips with real object orientation (as opposed to the named hashes of PHP) and am actually taking the leap into error handling, which, if you have dealt with any of the myriad bugs in any of my other projects, you’d know has been a bit of a foreign concept to me.

    Python project #2 is known as RepoMan (thanks to Ed Summers for the name). It attempts to solve a problem that not one but two other opensource projects already have solved admirably (I’ll go into more about this in a bit). RepoMan is an OAI Repository indexer that makes said repository available via SRU. I created it in an attempt to make our DSpace implementation searchable from remote applications (namely, the site search and the upcoming alternative opac). It’s an extremely simple two script project that has only taken a week to get running largely due to the existence of two similar and available python scripts that I could modify for my own use. It’s also due to the help of Ed Summers and Aaron Lav.

    The harvester is, basically, Thom Hickey’s one page OAI harvester with some minor modification. I have added error handling (the two lines I added to compensate for malformed xml must have been over the “one page limit”) and instead of outputting to a text file, it shoves the records in a Lucene index (thanks to PyLucene). This part still needs some work (I’m not sure what it would do with an “updated” record, for example), but it makes a nice index of the Dublin Core fields, plus a field for the whole record, for “default” searches. This was a good exercise for me to work with xml, Python and Lucene, because I was having some trouble when trying to index the MODS records for the alternative opac.

    The SRU server is, basically, Dan Chudnov‘s SRU implementation for unalog. It needed to be de-Quixotefied and is, in fact, much more robust than Dan’s original (of course, unalog’s implementation doesn’t need to be as “robust”, since the metadata is much more uniform), but certainly having a working model to modify made this go much, much faster. The nice part is that there might be some stuff in there that Dan might want to put back into unalog.

    So, here is the result. The operations currently supported are explain and searchRetrieve and majority of CQL relations are unsupported, but it does most of the queries I need it to do and, most importantly, it’s less than a week old.

    So the burning question here is: why on earth would I waste time developing this when OCKHAM’s Harvest-to-Query is out there, and, even more specifically, OCLC’s SRW/U implementation for DSpace is available? Further, I knew full well that these projects existed before I started.

    Lemme tell ya.

    Harvest-to-Query looked very promising. I began down this road, but stopped about halfway down the installation document. Granted, anything that uses Perl, TCL and PHP has to be, well, something… After all, those were the first three languages I learned (and in the same order!). Adding in IndexData’s Zebra seemed logical as well since it has a built-in Z39.50 server. Still, this didn’t exactly solve my problem. I’d have to install yazproxy, as well, in order to achieve my SRU requirement. Requiring Perl, TCL, PHP, Zebra and yazproxy is a bit much to maintain for this project. Too many dependencies and I am too easily distracted.

    OCLC’s SRW/U seemed so obvious. It seemed easy. It seemed perfect. Except our DSpace admin couldn’t get it to work. Oh, I inquired. I nagged. I pestered. That still didn’t make it work. I have very limited permissions on the machine that DSpace runs on (and no permissions for Tomcat), so there was little I could do to help. This also solved a specific purpose, but didn’t necessarily address any other OAI providers that we might have.

    So, enter RepoMan. Another wheel that closely resembles all the other wheels out there, but possibly with minor cosmetic changes. Let a thousand wheels be invented.

  • Implementing COinS in the SFX Menu

    Since Andrew asked:

    This should go in your banner.tmpl file. I am not bothering with BANNER_TYPE_3, since I am not really sure how I would represent that in a COinS format, anyway.

    Thanks to David Walker for showing me how this would work.

    <TMPL_IF BANNER_TYPE_1>
<TMPL_IF ATITLE>
<TR valign="top">
<TD class="LabelBold">Title:</TD>
<TD class="Label"><TMPL_VAR NAME="ATITLE" ESCAPE='HTML'>
</TD>
</TR>
</TMPL_IF>
<TR valign="top">
<TD class="LabelBold">Source:</TD>
<TD class="Label">
<TMPL_IF BTITLE>
<TMPL_VAR NAME="BTITLE" ESCAPE='HTML'>
</TMPL_IF>
<TMPL_IF ISBN>
[<TMPL_VAR NAME="ISBN" ESCAPE='HTML'>]
</TMPL_IF>
<TMPL_IF AULAST>
<TMPL_VAR NAME="AULAST" ESCAPE='HTML'>
</TMPL_IF>
<span class="Z3988" title="ctx_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:book<TMPL_IF ATITLE>&rft.atitle=<TMPL_VAR NAME="ATITLE" ESCAPE='URL'></TMPL_IF><TMPL_IF BTITLE>&rft.title=<TMPL_VAR NAME="BTITLE" ESCAPE='URL'></TMPL_IF><TMPL_IF ISBN>&rft.isbn=<TMPL_VAR NAME="ISBN" ESCAPE='URL'></TMPL_IF><TMPL_IF AULAST>&rft.aulast=<TMPL_VAR NAME="AULAST" ESCAPE='URL'></TMPL_IF><TMPL_IF INST>&rft.inst=<TMPL_VAR NAME='INST' ESCAPE='URL'></TMPL_IF><TMPL_IF YEAR>&rft.date=<TMPL_VAR NAME='YEAR' ESCAPE='URL'></TMPL_IF><TMPL_IF SPAGE>&rft.spage=<TMPL_VAR NAME='SPAGE' ESCAPE='URL'></TMPL_IF><TMPL_IF EPAGE>&rft.epage=<TMPL_VAR NAME='EPAGE' ESCAPE='URL'></TMPL_IF>"></span>
</TD>
</TR>
<TR>
<TD>
<TMPL_IF INST>
<TMPL_VAR NAME='INST' ESCAPE='HTML'>
</TMPL_IF>
<TMPL_IF YEAR>
yr:<TMPL_VAR NAME='YEAR' ESCAPE='HTML'>
</TMPL_IF>
<TMPL_IF SPAGE>
pg:<TMPL_VAR NAME='SPAGE' ESCAPE='HTML'>
</TMPL_IF>
<TMPL_IF EPAGE>
-<TMPL_VAR NAME='EPAGE' ESCAPE='HTML'>
</TMPL_IF>
</TD>
</TR>
</TMPL_IF>

<TMPL_IF BANNER_TYPE_2>
<TMPL_IF ATITLE>
<TR valign="top">
<TD class="LabelBold">Title:</TD>
<TD class="Label"><TMPL_VAR NAME="ATITLE" ESCAPE='HTML'></TD>
</TR>
</TMPL_IF>
<TR valign="top">
<TD class="LabelBold">Source:</TD>
<TD class="Label">
<TMPL_IF JTITLE>
<TMPL_VAR NAME="JTITLE" ESCAPE='HTML'>
</TMPL_IF>
<TMPL_IF ISSN>
[<TMPL_VAR NAME="ISSN" ESCAPE='HTML'>]
</TMPL_IF>
<TMPL_IF AULAST>
<TMPL_VAR NAME="AULAST" ESCAPE='HTML'>
</TMPL_IF>
<TMPL_IF YEAR >
yr:<TMPL_VAR NAME='YEAR' ESCAPE='HTML'>
</TMPL_IF>
<TMPL_IF VOLUME >
vol:<TMPL_VAR NAME='VOLUME' ESCAPE='HTML'>
</TMPL_IF>
<TMPL_IF ISSUE >
iss:<TMPL_VAR NAME='ISSUE' ESCAPE='HTML'>
</TMPL_IF>
<TMPL_IF SPAGE>
pg:<TMPL_VAR NAME='SPAGE' ESCAPE='HTML'>
</TMPL_IF>
<TMPL_IF EPAGE >
-<TMPL_VAR NAME='EPAGE' ESCAPE='HTML'>
</TMPL_IF>
<span class="Z3988" title="ctx_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:journal<TMPL_IF ATITLE>&rft.atitle=<TMPL_VAR NAME="ATITLE" ESCAPE='URL'></TMPL_IF><TMPL_IF JTITLE>&rft.jtitle=<TMPL_VAR NAME="JTITLE" ESCAPE='URL'></TMPL_IF><TMPL_IF ISSN>&rft.issn=<TMPL_VAR NAME="ISSN" ESCAPE='URL'></TMPL_IF><TMPL_IF AULAST>&rft.aulast=<TMPL_VAR NAME="AULAST" ESCAPE='URL'></TMPL_IF><TMPL_IF INST>&rft.inst=<TMPL_VAR NAME='INST' ESCAPE='URL'></TMPL_IF><TMPL_IF YEAR>&rft.date=<TMPL_VAR NAME='YEAR' ESCAPE='URL'></TMPL_IF><TMPL_IF SPAGE>&rft.spage=<TMPL_VAR NAME='SPAGE' ESCAPE='URL'></TMPL_IF><TMPL_IF EPAGE>&rft.epage=<TMPL_VAR NAME='EPAGE' ESCAPE='URL'></TMPL_IF><TMPL_IF VOLUME >&rft.volume=<TMPL_VAR NAME='VOLUME' ESCAPE='URL'></TMPL_IF><TMPL_IF ISSUE >&rft.issue=<TMPL_VAR NAME='ISSUE' ESCAPE='URL'></TMPL_IF>"></span>
</TD>
</TR>
</TMPL_IF>

<TMPL_IF BANNER_TYPE_3>
<TR valign="top">
<TD class="LabelBold">Source:</TD>
<TD class="Label">
<TMPL_IF TITLE>
<TMPL_VAR NAME="TITLE" ESCAPE='HTML'>
</TMPL_IF>
<TMPL_IF NUMBER>
[<TMPL_VAR NAME="NUMBER" ESCAPE='HTML'>]
</TMPL_IF>
<TMPL_IF INVLAST >
<TMPL_VAR NAME='INVLAST' ESCAPE='HTML'>
</TMPL_IF>
<TMPL_IF YEAR >
yr:<TMPL_VAR NAME='YEAR' ESCAPE='HTML'>
</TMPL_IF>
</TD>
</TR>
</TMPL_IF>

  • Abstract Artunit

    I’ve mentioned several times in this space the OPAC redesign project that Art and I are working on. There hasn’t really been anything to show, to date, because it’s taken a very long time to get actually get the data out of Voyager. There are easier and faster ways we could have done this, probably, but we’ve been a little bogged down trying to get this to work in Art’s webdav environment. This has required sucking the data out of Oracle according to LCC and that’s been no easy task. GovDocs are in a different hierarchy, based on SUDOC (CODOC, for Art).

    In the meantime, I get emails from Art at 12:30 at night, 7:30 in the morning that say things like:

    I am woefully weak on python but I know you have been working with python lately and I wondered if the approach I am using makes sense. I am persisting date modified information with a python shelve. So it looks like:

    shelf[url] = last_modified

    This seems to work wonderfully, but I needed to add:

    import dumbdbm

    for the shelf to have somewhere to put the info. What I think is supposed to happen is that the shelf command looks for some sort of database option and cycles through them all looking for storage. The “import dumbdbm” seems to be a way to add an option if no other is found. Have you ever tried anything like this? I wanted to use pickle/cpickle but a million links would probably throttle it.

    … I, of course, have no idea what he’s talking about, but it’s flattering nonetheless that he thinks I might.

    Anyway, last week I started actually working with PyLucene and our metadata mirror files (Art, meanwhile, is doing similar work with Cocoon/Lucene) and I came across what is possibly the most useful byproduct of this project. While I was preparing the logic to frbrize the mirror data, it struck me that it doesn’t have to be perfect, at first.

    By separating the data from the ILS, we can create any kind of interface we want, indeed several, should we choose, without worrying about affecting the backend system at all. We can combine records, add metadata as necessary, remove it if it doesn’t work properly, tweak our search algorithms, and incorporate it into any sort of system we want, because it would have absolutely no effect on the ILS itself. We’ll still have the original “authority” should we mess anything up too badly and we’ll have all kinds of value that couldn’t (and probably shouldn’t) go in a “conventional opac”.

    This sort of abstraction from the “inventory control system” is such a basic programming principle that I have to wonder why no vendors implement it (even I, as an untrained hacker understand the importance of this). It also abstracts the user interface from the catalogers a bit — added bonus. Catalogers are great for many things, but designing user interfaces generally isn’t one of them.