At the beginning of the month, I gave two presentations at the Ontario Library Association’s SuperConference. I had a good time. My first time to Toronto and it snowed. Although, with all due respect, Mr. Lee, YYZ totally sucked.
My presentations were:
Librarian’s Lib: Taking control of what’s ours
The Communicat: contribute to the collective collection, comrade
I have been working on Fancy-Pants quite a bit in the last couple of weeks. This is an AJAX layer over Voyager’s WebVoyage — an attempt to de-suck-ify its interface a bit. Why is it called Fancy-Pants? Well, Voyager still has the same underwear, it’s just got a new set of britches.
There are two main problems that it’s trying to solve:
Now, we’re already doing the second one, but it’s not implemented particularly well. While we were solving those problems, we wanted to see what we could do about that god-awful table based display.
I took NCSU’s Endeca layout as the baseline template for what I wanted the results to look like. Right now, Fancy-Pants can only be accessed via this Greasemonkey script [get Greasemonkey here]. Greasemonkey, of course, wouldn’t be a requirement, but we’re using it to inject the initial javascript call since we’re having to work on a live system.
For the title list screen, the javascript is looping through the bib ids on the page (it grabs them from the ‘save record’ checkboxes) and sends them to a Ruby on Rails app that queries Voyager’s Oracle database and builds a new result set. The javascript hides the original page results (display: none) and inserts a div with the new results. If there are multiple 856es or locations, the result has expanding/collapsing divs to show/hide them.
I send the query terms to Yahoo’s spell check API and will return a link to any suggestions it gives. No, this isn’t the ideal, but I’m still in proof-of-concept stage.
Things I still want to do with title list screen are:
I’m currently not doing much with the record view page, but I am adding a direct link to the record. I plan on integrating Umlaut responses here, as well as other context sensitive items – especially those that don’t conform well to OpenURL requests.
If you were able to install the Greasemonkey script and want to try it out, go to GIL’s keyword search and try:
Also try a journal search for “Nature”. Then try whatever floats your boat and let me know how it worked. If you notice that it’s really slow, this is actually because of Voyager. The “Available online” and relevance icons are all rendered dynamically and they just grind the output to a halt. When we go live with this, we’d disable those features in WebVoyage to speed things up.
Fancy-pants is by no means a final product. I view this as a bridge between what we have and an upcoming Solr based catalog interface. The Solr catalog will still need to interface with Voyager, so Fancy-pants would transition to that. Ultimately, I would like this whole process to eventually lead to the Communicat.
I have a very conflicted relationship with our archives department. While their projects still need to get done, their services get very little use (especially when compared to other pending projects) and every time I get near any of their projects, it starts to become “Ross Singer and the Tar Baby”. Everything, EVERY SINGLE THING, archives/special collections has ever created is arcane, dense, and enormously time consuming. It always seems simple and then it somehow always turns into EAD.
About a year ago, I was tasked with developing something to help enable archives publish their recently converted EAD 2002 finding aids to HTML. I knew that XSLT stylesheets existed to do the heavy lifting in this so I thought it wouldn’t be too hard to get this up and running.
But it never works out that way with archives. The stylesheets that did what they liked worked only in Saxon and Saxon only works with Java (well, now with .Net, too — but that still didn’t help at the time). I wasn’t going to take on a language that I only poke at with a long stick on the most ambitious of days for some archives project (no offense, archives, but come on…). My whining caught Jeremy Frumkin‘s attention who pointed me at a project that Terry Reese had done. It was a PHP/MySQL project that took the EAD, called Saxon from the commandline and printed the resulting HTML. It also indexed all these parts of the finding aid and put them in a MySQL database to enable search. I could never get the search part to work very well with our data, so I gave up that part, focusing instead on the Saxon->XSLT->cache to HTML part.
I set up a module in our intranet that let the archivists upload xml files, preview the result with the stylesheet, rollback to an earlier version, etc. No matter what, though, this was a hack. And, most importantly, I never got search working. Also, the detailed description (the dsc) was incredibly difficult to get to display how archives wanted it.
Another thing that nagged at me was, for all this work I (and they) had invested in this, how was this really any better than the HTML finding aids they were migrating from? They put all this work into encoding the EAD and all that we were really doing was displaying a web page that was less flexible in its output than the previous HTML.
This summer, I met with archives to discuss how we were going to enable searching. Their vision seemed pretty simple to implement.
Why do I keep falling for that? How many hours had I already invested in something I thought would be trivial?
The new system was (of course) built in Rails. The plan was to circumvent XSLT altogether so:
The fulltext indexing would be provided by Ferret. I thought I could hammer this out in a couple of weeks. I think dumb things like this a lot.
The infrastructure went up pretty quickly. Uploading, indexing, and displaying the browse lists (up until now, they had to get the web designer to add their newly encoded finding aid to the various pages to link to it) all took a little over a week, maybe. Search took maybe another week. For launch purposes, I wasn’t worried about pagination of search results. There were only around 210 finding aids in the system, so a search for “georgia” (which cast the widest net) didn’t really make a terribly unmanageable results page. That’s the nice thing about working with such a small dataset that’s not heavily used. Inefficiencies don’t matter that much. I’ve since added pagination (took a day or two).
No, like last time, the real burden was displaying the finding aid. My initial plan, parsing the XML and divvying it up into a bunch of Ruby objects, was taking a lot of time. EAD is inordinately difficult to put into logical containers that have easily usable attributes. That’s just not how EAD rolls. I found I was severely delaying launch trying to shoehorn my vision on the finding aid, which archives didn’t really care about, of course. They just wanted searching. So, while I was working out my EAD/Ruby object modeler, I deferred to XSLT to get the system out the door.
Rather than using Saxon this time, I opted for Ruby/XSLT (based on libxml/libxslt) for main part of the document and ruby scripting/templates for the detailed collection list. The former worked pretty well (and fast!) but the latter was turning into a nightmare of endless recursion. When I tried looping through all of the levels (EAD can have 9 levels of recursion, c01-c09, starting at any number — I think — and going to any depth), my vain attempts either showed the horrid performance of REXML (a native Ruby XML parser) or attempts at navigating the recursion that would leave you clutching at the fibers of sanity that remain when you get this far in an EAD document.
Finally I found what I thought was my answer: a nifty little Ruby library called CobraVsMongoose that would transform a REXML document to a Ruby hash (or vice-versa). It was unbelievably fast and made working with this nested structure a WHOLE lot easier. There was some strangeness to overcome (naturally). For instance, if an element’s child nodes include an element that appears more than once, it will nestle the the children in an array. If there are no repeating elements, it will put then child nodes in a Hash, (or the other way around, I can’t remember) so you have to check what the object is and process it differently accordingly. Still, it was fast, easy to manipulate and allowed me to launch the finding aids system.
Everybody was happy.
And then archives actually looked at the detailed description. Anything in italics or bold or in those weird ’emph’ tags wasn’t being displayed. Ok, no problem, I just need to mess with the CobraVsMongoose function… oh wait. Yeah, what I hadn’t really thought about was that Ruby hashes don’t preserve sort order, so there was no way to get the italicized or bolded or whatevered text to display where it was supposed to in the output.
Damn you, tarbaby!
Back to the drawing board. I decided to return to REXML, emboldened now by (what I thought) was a better handle on the dsc structure and some better approaches to the recursion. Every c0n element would get mapped to a Ruby object (Collection, Series, Subseries, Folder, Item, etc.) and nest as children to each other and have partial views that would display them properly.
On my first pass, it was taking 28 seconds for our largest finding aid to display. TWENTY EIGHT SECONDS?! I would like to note, as finding aids go, ours aren’t very large. So, after tweaking a bit more (eliminating checking for children in Item elements, for example), I got it down to 24 seconds. Still not exactly the breathtaking performance I had hoped for.
What was bugging me was how quick CobraVsMongoose was. Somehow, despite also using REXML, it was fast. Really fast. And here my implementation seemed more like TurtleVsSnail. I was all set to turn to Libxml-Ruby (which would require a bunch of refactoring to migrate from REXML) when I found my problem and its solution. This was last night at 11PM.
While poring over the REXML API docs, I noticed that the REXML::Element.each_element method’s argument was called ‘xpath’. Terry had written about how dreadfully slow XPath queries were with REXML and, as a result, I thought I was avoiding them. When I removed the path arg from the each_element call in one of my methods and just iterated through each child element to see if its name matched, it cut the processing time in half! So, while 12 seconds was certainly no thoroughbred, it was definitely the right track. When I eliminated every xpath in the recursion process, I got it down to about 5 seconds. Add a touch of fragment caching and the natural performance boost of a production vs. development site in rails, and I think we’ve got a “good enough for now” solution.
There’s a little more to do with this project. I plan on adding an OpenSearch interface (I have all the plumbing from adding it to the Umlaut), an OAI provider and an SRU interface (when I finally get around to porting that CQL parser). And, yeah, finishing the EAD Object Model.
But right now, archives and I need to spend a little time away from each other.
In the meantime, here’s the development site.
Library Geeks #4 is out. I’m back (instead of being the ‘Poltergeek’ of episode 3), and Dan set this one up in kind of a neat way. He certainly leads and guides the dialogue, but it’s much more of a roundtable and informal discussion. No doubt this is largely due to the fact we’re all pretty good friends. Still, I learned a lot about Ed that I didn’t know — pretty amazing since we hang out every day and work on so much together.
Bobby McFerrin, eat your heart out.
I have been thinking a lot this week about how libraries are skirting ever closer to a precipice they generally refuse to acknowledge.
While certainly a cynic, I’m not generally a Cassandra, so before proclaiming last rites on the library, I wanted to make sure I had thought about this some. This was probably spurred on by the “Murder MARC” thread on NGC4Lib.
And I got to thinking about travel agents.
Fifteen or twenty years ago, it was nigh onto impossible to book a trip anywhere without a travel agent. My mother worked part-time for a travel agent; travel agents were everywhere. It seemed as much a part of travelling as does a real estate agent seem part of home buying today. It’s technically possible to work without them, but life is going to be a whole lot easier if you do. Travel agents held the keys to your vacation; you were at the mercy of where, when and how much your vacation was to be.
Then, sometime during the dot-com blitz, up popped sites such as Expedia, Travelocity and Orbitz. Suddenly, the consumer controlled his or her own travel. The traveller could waste as much time as he or she wanted looking for the perfect getaway; setting up fare watches to places they might randomly want to go; researching the hotel on the receiving end. All this without hassle of somebody trying arrange the trip for them. Sure, you might not have found the best rate or gotten a free meal thrown in at your destination, but the merits outweighed the costs. You began to know fewer and fewer travel agents.
For a long time librarians viewed Amazon.com and the other online booksellers as their primary competition. It made sense; both parties were hawking roughly the same physical wares. For a very long time, booksellers and libraries have coexisted. There was an understanding of the differences between the two models and, despite attempts to make the library catalog act more like Amazon, the market for each seemed fairly well intact.
Search engines, however, were happy to freely give away information indiscriminently. As the Googles and Yahoos indexed more and more content and the information grew freer and freer, libraries began to seem more antiquated and backwards in comparison. With more and more information available — preprints, Open Access, book digitization efforts — the need for specific content began to wane. Users began to lose interest working with arcane brokers of information when the information they were finding on their own met their needs. Librarians, in response, chastized the users’ choices and maintained that a steady diet of hard work and piety was the only way to achieve scholarly salvation. Dogmatic adherence to preservation of metadata took precedence over improving the user’s ability navigate and discover. After all, we know better than the user how to find the best fares, right?
I’m not a Cassandra, I’m really not. Disruptive technologies can lead to revolutionary changes, after all.
However, these thoughts came to a boil today. I have been given a lot of criticism about the way I rely on the search engines to provide a lot of functionality in the Ümlaut. How, despite the money we pay for our licensed resources, I still prominently display free web search results.
The fact of the matter is that it’s impossible for me include our precious resources for anything useful, while including Google, Yahoo and Amazon was a piece of cake. The best I can hope to give our users from our collection are links to lists of databases or subject guides; without considerable work on my part and the librarians all I can hope to get them is a generic link to all of our databases or a list of subject guides. With the search engines I can at least narrow down the information as to be somewhat useful. And if it’s not? They don’t have to use it. There’s a link to the library, that’s one link away from the databases or subject guides.
At this point it’s important for us to empower the user; not with arcane searching techniques in hard to find resources, but by leveraging our systems so they can be integrated easily into wherever the user is looking and by exposing our content and services in ways that fit into the user’s sphere of control and comfort. If we don’t, the user will still find information that is ‘good enough’ for their needs and more will be added every day. Then, we’ll not only have lost our control, but our relevance as well.
I am embarking on an experiment to learn the value of an ALA-accredited MLS.
I’ll give details in a couple months when it’s over.
Ariadne #48 includes “Introducing unAPI“, written by Dan, Jeremy, Peter, Michael, Mike, Ed and me. It explains the potential of unAPI and shows some examples on implementations.
Like the last article I “wrote” with this group, I didn’t have to do much. On the flipside, publishing has such a minute effect (if any) on my career, I guess the “effort” reflects the “rewards”.
Still, this one was tough. A refactoring of the Umlaut broke the unAPI interface right about the time the editors were going over the example cases in the article. It was extremely difficult to put development cycles into this when I had a ton of other things I desperately needed to get completed before the start of the semester.
And that’s where the struggle is. While I’m completely sold on the “potential” of unAPI, that’s all it is right now. COinS was much easier concept for me get behind (although I would definitely say I have invested more in unAPI than COinS) since it’d be so much easier to implement on both the information provider and client ends. UnAPI is a much, much more difficult sell. However, the payoff is so much greater, one has to have faith and sometimes dedicate time that should probably have been spent on more immediate needs.
Dan Chudnov “interviewed†me about the Ãœmlaut for the first “Library Geeksâ€. It’s crude and show obvious signs of “learning as we’re goingâ€, but if you want to hear more about the Ãœmlaut from my voice and hear my favorite curse word (and boy is it a doozy), take a listen.
Note the “3-2-1″ at around the 31st minute. It took us several attempts to get this recorded. The first session was lost after about 30 minutes of recording courtesy of GarageBand crashing. We then lost the 2nd quarter, again thanks to GarageBand crashing. I had to do the 10 questions twice. Why? I’ll let you guess.
In other Ümlaut related news (is there any other right now?), I have the project page up. I’ll be adding to the HowUmlautWorks when I get free moments.
Also, thanks to Dan’s lead (lots of links to One Big Library today), I added the OCLC Resolver Resolver Registry to the Ümlaut. When you start a session, it checks your IP against the OCLC registry. If the registry returns a link resolver based on your IP, the Ümlaut will check to see if your resolver is SFX (I’ll be happy to add other XML enabled resolvers) and, if so, includes those holdings in the results, as well. If not, it includes a link to your resolver. If it doesn’t work for you, my guess is that OCLC has ILLiad (or whatever you use for Interlibrary Loan) first in your library’s profile.
Here are some screenshots of this in action:
How this citation looks at Georgia Tech [pdf]
How this citation looks at Drexel [jpg]
Thanks to Gabriel Farrell for trying this at Drexel and supplying the screenshot (and congratulations on the new job!). It still needs some work (for instance, it needs to tell the user that the fulltext is courtesy of Drexel or wherever), but I think it’s a start.
We’re about a week away from launch, so I’ll be working on migrating to the production server. I’ll be sure to document it so others can partake if they desire.
It’s been a while since I’ve written anything about the ümlaut. It’s been a while since I’ve written about anything, really. Lots of reasons for that: been frantically trying to pull the ümlaut together in time to launch for fall semester, and I’ve got this little bit of business going on…
Still, it’s probably time to touch on some of the changes that have happened.
The intial design was… shaky… at best. While the new backend is probably still shaky (it is, after all, my creation), it’s certainly more thought-out. Incoming requests are split by their referent and referrers (see Thom Hickey’s translation for these arcane terms) and the referent is checked against a ‘Ferret store’ of referents. The rationale here is that citations take a bunch of forms in regards to their completeness and quality, so we do some fulltext searching against an index of previously resolved referents to see if we’ve dealt with this before.
It then stores the referent and the response object as Marshaled data, which is great, except it royally screws up trying to tail -f the logs.
Heather King, our web designer here at Tech, has vastly improved the design. There’s still quite a bit more to do (isn’t there always?), but we’ve got a good, eminently usable, interface to build upon. The bits that need to be cleaned up (mainly references to other citations) won’t be that hard to clean up.
Well, read-only support. Connotea, Yahoo’s MyWeb and Unalog support were pretty easy to add courtesy of their existing APIs. The downside is that I can only hope to find bookmarks based on URL which… doesn’t work well. I really wish Connotea would get some sort of fielded searching going on. Del.icio.us support, which would be great, can’t really happen until they ease the restrictions on how often you can hit it.
CiteULike was a bit more of a hack, as it has no API. Instead, I am finding CiteULike pages in the Google/Yahoo searches I was already doing, grabbing the RSS/PRISM and tags and then doing a search (again, retrieving the RSS/PRISM) with the tags in the first article. It’s working pretty well, although I need to work out title matching since both Yahoo and Google truncate HTML titles. I plan on adding the CiteULike journal table of contents feeds this way, too.
Let’s face it. The ümlaut was bloody slow. There were a couple of reasons for this, but it was mainly due to hitting Connotea and trying to harvest any OAI records it could find for the given citation. The kicker was that this was probably unnecessary for majority of the users. Now we’ve pushed the social bookmarkers and the OAI repositories to a ‘background task’ that gets called via javascript when the page renders. It’s not technically AJAX as much as a remote procedure call, but AJAX is WebTwoFeriffic! Besides, this is a Rails project. Gotta kick up the buzz technologies.
The ümlaut now grabs subject headings from Pubmed (MeSH); LCSH from our catalogs; SFX subjects; tags from Connotea, CiteULike, MyWeb and unalog; and subjects from the OAI repositories and stores them with the referent. It also stores all of these in the Ferret store. The goal here is to search on subject headings to find recommendations to other similar items. As of this writing, there is only one citation with subject associations, so there’s nothing really to see here.
The big todo that’s on my plate for the rest of the week is adding statistics gathering. I’ve got my copy of An Architecture for the Aggregation and Analysis of Scholarly Usage Data by Bollen and Van de Sompel printed out and I plan on incorporating their bX concept for this.
This is really the first time I’ve gotten to sit down and write since I got back from the first annual ELUNA (Ex Libris Users, North America) Conference earlier this week in Knoxville, TN. I made a presentation on the umlaut (and I’ll go into that a bit later), but, in general, I really enjoy the Ex Libris user’s group conferences. In fact, it’s an amazing contrast to Endeavor’s EndUser, the only other user’s group conference I’ve ever been to.
ELUNA combined NAAUG (North American Aleph User’s Group) and SMUG (SFX/Metalib User’s Group) which historically happened one after the other (and was becoming problematic since Ex Libris was introducing more products). This had positive and negative effects: it’s good that Aleph/SFX users didn’t have to spend an entire week in conference, but the concurrent track sessions in the concurrent tracks meant that I felt I missed a lot. Also, Aleph tended to dominate the non-track-dependent sessions. The hour and a half long sessions also meant that there weren’t very many, so I feel like I only got to take in 2 non-vendor presented sessions (outside my own). Still, I don’t think I really have any complaints.
The first (real) session I attended was a panel discussion about the SFX API. There was nothing particularly “ground breaking” here for me (since I’m pretty familiar with the API), but it did show some really clever applications taking advantage of the API (all three of which I can see trying to implement at Tech). The first was Anne Highsmith of Texas A&M. She was using the LinkFinderPlus feature in Voyager to create SFX links in the OPAC. They’re doing some clever work in making sure there’s electronic holdings before actually displaying the link. The added bonus to this approach is the lack of need to populate 856 tags. I think this is a really good avenue to implement link resolving in our state union catalog: utilize the LinkFinderPlus feature to point to a web service that asks the OCLC Resolver Service if the user’s IP fits any institution (and, if not, allow them to associate themself with an institution, either through the OCLC Service or utilizing some variation of the Rails Resolver Router on Rails) and display links to their appropriate resolver (which would help institutions like Emory).
Next up was Margery Tibbets of the CDL. They are using the API to LinkOut items in Pubmed. To me this seemed like a clever way to deal with the annoyances of OpenURL enabling Pubmed.
Mark Dehmlow of Notre Dame then described a process of doing document delivery of a package of Springer journals that they had cancelled subscribing to via Springer Express. They’ve duplicated the Springer targets and removed all threshold information, so it’s always available. If the user requests something they have access to (or predates Springer coverage completely), they give options to their existing Springer access or ILL accordingly. If it’s a Springer Title from 2006 onward, they present a link for the library to purchase it through Springer Express. Since it’s just 2006 titles, they don’t have enough data yet to see how successful it really is (or how much it will wind up costing in the long run).
After lunch, I was up. All in all, I think the umlaut was pretty well received and it was nice to be introduced by my old boss, Bill Britten. In fact, it was really great to see all my old coworkers. As usual, I geeked out a little too much and used too many acronyms and generally talked at too advanced a level. This is something I’ve got to start working on. One of the problems is that I feel I’m not explaining the umlaut very well. While, yes, it is a replacement for the SFX menu the point is more about eliminating the need for this weird piece of software that sits between them (the database vendors) and us. Plus, it’s actually trying to analyze what’s coming in and what the user can access (whether it’s licensed content or not and from any number of places the user can access things) and eliminate the need for them to look in multiple places for it.
Both John Webb and Sean Chen brought up the potential negative of this approach: “You’re making a decision to just cram more stuff in the menu”. I think it would have helped if I could have gotten the “Fancy Pants” implementation (Fancy Pants is what we’re calling our redesign of WebVoyage, done completely in AJAX) working in time.
After my presentation, Oren Beit-Arie gave a presentation that mainly featured Primo. Primo definitely shows promise, but it’s going to be a hard sell for Tech (mainly financially). Also, there’s a part of me that feels that it could be mostly accomplished with Solr (otherwise I’m going to be wasting a considerable amount of Tech’s time in the upcoming months).
That night they had barbecue, free beer and live music by R.B. Morris and Hector Qirko in the World’s Fair Park. I sure do love free beer.
Tuesday morning was spent in vendor sessions (SFX and Metalib). The biggest news here was that Metalib 4 will include clustering courtesy of Vivisimo.
After a wonderful lunch at the Tomato Head and a brutal Q&A session with the Ex Libris leadership (I now know more about Aleph than I ever cared to know- none of it terribly positive), I went to a session about Windows Live Academic Search by MSN’s own Mike Buschman. This was a good session (even if Mike was plagued with technical difficulties): he acknowledged the problems with the AJAX approach (like complete lack of Safari support), and issues regarding OpenURL linking. Still, I admire MSN’s take on this (transparency of what’s being indexed, limited scope, indexing scholarly content specifically, as opposed to filtering scholarly items out of the general index) and, if they can get the GUI working better, I think will have a pretty nice competitor to Google Scholar. The thought of an open, Academic-only API makes me excited, as well.
There’s nothing terribly interesting to report about Wednesday (mostly business stuff).
This was a good trip. I made some good contacts (hopefully some joint projects with the University of Rochester, possibly Yale), saw some interesting projects and new products, caught up with old friends (and some newer ones), and maintained my confidence in the way Ex Libris is run as a company. Can’t ask for much more than that out of a vendor conference.