Dilettante's Ball

Blog

  • Üpdate

    It’s been a while since I’ve written anything about the ümlaut. It’s been a while since I’ve written about anything, really. Lots of reasons for that: been frantically trying to pull the ümlaut together in time to launch for fall semester, and I’ve got this little bit of business going on…

    Still, it’s probably time to touch on some of the changes that have happened.

    1. The backend has been completely updated

      2. The intial design was… shaky… at best. While the new backend is probably still shaky (it is, after all, my creation), it’s certainly more thought-out. Incoming requests are split by their referent and referrers (see Thom Hickey’s translation for these arcane terms) and the referent is checked against a ‘Ferret store’ of referents. The rationale here is that citations take a bunch of forms in regards to their completeness and quality, so we do some fulltext searching against an index of previously resolved referents to see if we’ve dealt with this before.

      It then stores the referent and the response object as Marshaled data, which is great, except it royally screws up trying to tail -f the logs.

    2. New design

      3. Heather King, our web designer here at Tech, has vastly improved the design. There’s still quite a bit more to do (isn’t there always?), but we’ve got a good, eminently usable, interface to build upon. The bits that need to be cleaned up (mainly references to other citations) won’t be that hard to clean up.

    3. Added Social Bookmarking Support

      4. Well, read-only support. Connotea, Yahoo’s MyWeb and Unalog support were pretty easy to add courtesy of their existing APIs. The downside is that I can only hope to find bookmarks based on URL which… doesn’t work well. I really wish Connotea would get some sort of fielded searching going on. Del.icio.us support, which would be great, can’t really happen until they ease the restrictions on how often you can hit it.

      CiteULike was a bit more of a hack, as it has no API. Instead, I am finding CiteULike pages in the Google/Yahoo searches I was already doing, grabbing the RSS/PRISM and tags and then doing a search (again, retrieving the RSS/PRISM) with the tags in the first article. It’s working pretty well, although I need to work out title matching since both Yahoo and Google truncate HTML titles. I plan on adding the CiteULike journal table of contents feeds this way, too.

    4. Improved performance thanks to the magic of AJAX

      5. Let’s face it. The ümlaut was bloody slow. There were a couple of reasons for this, but it was mainly due to hitting Connotea and trying to harvest any OAI records it could find for the given citation. The kicker was that this was probably unnecessary for majority of the users. Now we’ve pushed the social bookmarkers and the OAI repositories to a ‘background task’ that gets called via javascript when the page renders. It’s not technically AJAX as much as a remote procedure call, but AJAX is WebTwoFeriffic! Besides, this is a Rails project. Gotta kick up the buzz technologies.

    5. Now storing subjects in anticipation of recommender service

      6. The ümlaut now grabs subject headings from Pubmed (MeSH); LCSH from our catalogs; SFX subjects; tags from Connotea, CiteULike, MyWeb and unalog; and subjects from the OAI repositories and stores them with the referent. It also stores all of these in the Ferret store. The goal here is to search on subject headings to find recommendations to other similar items. As of this writing, there is only one citation with subject associations, so there’s nothing really to see here.

    The big todo that’s on my plate for the rest of the week is adding statistics gathering. I’ve got my copy of An Architecture for the Aggregation and Analysis of Scholarly Usage Data by Bollen and Van de Sompel printed out and I plan on incorporating their bX concept for this.

  • Why Dan is wrong (and, yet, totally right)

    Dan has written (twice now) about the ‘ILS Bill of Rights’ and its complete lack of perspective.

    He’s right in his equation (or lack thereof) of the OPAC to “Free Speech” and “lack of Government Oppression regarding which religion I choose”; however, I disagree with him about the ability to be able to freely change vendors at will when the vendor doesn’t meet our needs (which means we’d be changing vendors, like, weekly).

    That being said… why bother with the ILS?

    Art Rhyno and I have been trying for more than a year to extract our bib database into a Lucene index to work with as we please. It took until NCSU released their Endeca index for us to be taken even remotely seriously. Still, the question remains… why bother with the ILS? If we can change the public interface, are the staff interfaces so lacking? Do we need to AJAX-ify the cataloging client or the circulation interface?

    Dan’s right, we don’t need an ILS “Bill of Rights”, but we do need to fix our problems. And we don’t have to change vendors to do so — we just need a bit of vision.

  • ELUNAmating

    This is really the first time I’ve gotten to sit down and write since I got back from the first annual ELUNA (Ex Libris Users, North America) Conference earlier this week in Knoxville, TN. I made a presentation on the umlaut (and I’ll go into that a bit later), but, in general, I really enjoy the Ex Libris user’s group conferences. In fact, it’s an amazing contrast to Endeavor’s EndUser, the only other user’s group conference I’ve ever been to.

    ELUNA combined NAAUG (North American Aleph User’s Group) and SMUG (SFX/Metalib User’s Group) which historically happened one after the other (and was becoming problematic since Ex Libris was introducing more products). This had positive and negative effects: it’s good that Aleph/SFX users didn’t have to spend an entire week in conference, but the concurrent track sessions in the concurrent tracks meant that I felt I missed a lot. Also, Aleph tended to dominate the non-track-dependent sessions. The hour and a half long sessions also meant that there weren’t very many, so I feel like I only got to take in 2 non-vendor presented sessions (outside my own). Still, I don’t think I really have any complaints.

    The first (real) session I attended was a panel discussion about the SFX API. There was nothing particularly “ground breaking” here for me (since I’m pretty familiar with the API), but it did show some really clever applications taking advantage of the API (all three of which I can see trying to implement at Tech). The first was Anne Highsmith of Texas A&M. She was using the LinkFinderPlus feature in Voyager to create SFX links in the OPAC. They’re doing some clever work in making sure there’s electronic holdings before actually displaying the link. The added bonus to this approach is the lack of need to populate 856 tags. I think this is a really good avenue to implement link resolving in our state union catalog: utilize the LinkFinderPlus feature to point to a web service that asks the OCLC Resolver Service if the user’s IP fits any institution (and, if not, allow them to associate themself with an institution, either through the OCLC Service or utilizing some variation of the Rails Resolver Router on Rails) and display links to their appropriate resolver (which would help institutions like Emory).

    Next up was Margery Tibbets of the CDL. They are using the API to LinkOut items in Pubmed. To me this seemed like a clever way to deal with the annoyances of OpenURL enabling Pubmed.

    Mark Dehmlow of Notre Dame then described a process of doing document delivery of a package of Springer journals that they had cancelled subscribing to via Springer Express. They’ve duplicated the Springer targets and removed all threshold information, so it’s always available. If the user requests something they have access to (or predates Springer coverage completely), they give options to their existing Springer access or ILL accordingly. If it’s a Springer Title from 2006 onward, they present a link for the library to purchase it through Springer Express. Since it’s just 2006 titles, they don’t have enough data yet to see how successful it really is (or how much it will wind up costing in the long run).

    After lunch, I was up. All in all, I think the umlaut was pretty well received and it was nice to be introduced by my old boss, Bill Britten. In fact, it was really great to see all my old coworkers. As usual, I geeked out a little too much and used too many acronyms and generally talked at too advanced a level. This is something I’ve got to start working on. One of the problems is that I feel I’m not explaining the umlaut very well. While, yes, it is a replacement for the SFX menu the point is more about eliminating the need for this weird piece of software that sits between them (the database vendors) and us. Plus, it’s actually trying to analyze what’s coming in and what the user can access (whether it’s licensed content or not and from any number of places the user can access things) and eliminate the need for them to look in multiple places for it.

    Both John Webb and Sean Chen brought up the potential negative of this approach: “You’re making a decision to just cram more stuff in the menu”. I think it would have helped if I could have gotten the “Fancy Pants” implementation (Fancy Pants is what we’re calling our redesign of WebVoyage, done completely in AJAX) working in time.

    After my presentation, Oren Beit-Arie gave a presentation that mainly featured Primo. Primo definitely shows promise, but it’s going to be a hard sell for Tech (mainly financially). Also, there’s a part of me that feels that it could be mostly accomplished with Solr (otherwise I’m going to be wasting a considerable amount of Tech’s time in the upcoming months).

    That night they had barbecue, free beer and live music by R.B. Morris and Hector Qirko in the World’s Fair Park. I sure do love free beer.

    Tuesday morning was spent in vendor sessions (SFX and Metalib). The biggest news here was that Metalib 4 will include clustering courtesy of Vivisimo.

    After a wonderful lunch at the Tomato Head and a brutal Q&A session with the Ex Libris leadership (I now know more about Aleph than I ever cared to know- none of it terribly positive), I went to a session about Windows Live Academic Search by MSN’s own Mike Buschman. This was a good session (even if Mike was plagued with technical difficulties): he acknowledged the problems with the AJAX approach (like complete lack of Safari support), and issues regarding OpenURL linking. Still, I admire MSN’s take on this (transparency of what’s being indexed, limited scope, indexing scholarly content specifically, as opposed to filtering scholarly items out of the general index) and, if they can get the GUI working better, I think will have a pretty nice competitor to Google Scholar. The thought of an open, Academic-only API makes me excited, as well.

    There’s nothing terribly interesting to report about Wednesday (mostly business stuff).

    This was a good trip. I made some good contacts (hopefully some joint projects with the University of Rochester, possibly Yale), saw some interesting projects and new products, caught up with old friends (and some newer ones), and maintained my confidence in the way Ex Libris is run as a company. Can’t ask for much more than that out of a vendor conference.

  • A little dog, named ‘Snuggles’

    One of the cool things about having our TiVo hooked up our (really
    crappy) broadband connection (no thanks to you, Comcast) is the seemingly random shit that just comes to us.

    Yesterday I was, on a whim, flipping through the TiVo menus and I saw that I could subscribe to Rocketboom on it. On an average yesterday, I probably would have ignored this feature, since I would have had no idea what Rocketboom was. However, since Rocketboom was featured on either Morning Edition or Marketplace (since our crappy internet connection is out again, I can’t look it up to know) this week, I actually knew what it was and was interested in it.

    So tonight I got to watch my first Rocketboom. It featured Mark Hosler of Negativland and he mostly talked about copyright law (which makes sense coming from him). Negativland was heavily involved in the creation of the Creative Commons license. What I found interesting was that Mark Hosler was actually fairly critical of the CC license (for reasons I agree with).

    Since I started releasing things that other people might use, I have struggled with what license to release my work under. In the end, I have released most of my code under the MIT license (in my laziness, I’ve never actually bothered to learn the details of what this means — a coworker once told me it was the most relaxed of the OSI approved and I was happy to accept that explanation); I basically wanted an “I honestly don’t care what you do with this, just use it” license. I didn’t care how the code was reused. I didn’t even particularly care if I was attributed for it.

    Mark Hosler talked about the a la carte menu that CC gives the user for creating their own license terms (‘Cannot be used for profit’, ‘Cannot be changed’, etc.) and how this, ultimately, is ridiculous. Once released in the wild, the creator cannot expect to retain any control of his or her creation. It can be reconstituted, reused, recycled and remixed in ways that the creator never intended or expected and, if the original has any value, should be.

    What I liked about his viewpoint is that he noted a difference between the intellectual property and the artifact (at least, this is what I got from it). Negativland was compensated for making an original work that satisfied both their label and their audience. What becomes of the work (shy of outright plagiarism — that is to say, passing off the original work as one’s own) is none of Negativland’s business since they had an arrangement to be compensated to create the original work in the first place. His prevailing argument (again, in my opinion) is that intellectual property should be more defined by the inspiration, not the artifact.

    Unfortunately, I still don’t know what license to release my work under.

  • Indexing the OPAC to suit our users’ needs

    This may be the first implementation of the (proposed) openurl context set for cql.

    MARC field 490 $v, y’all.

    Granted, the openurl context set might not be approved (or might change radically), but I think this shows a compelling use case.

  • OMG! 1+z fu11 0f b10gg3rZ!

    The ‘biblioblogosphere’ expands exponentially.

  • Explosive Diaereses

    I’ve been waiting for a while to have this title. Well, actually, not a long while, and that’s testimony to how quickly I’m able to develop things in Rails.

    While I think SFX is fine product and we are completely and utterly dependent upon it for many things, it does still have its shortcomings. It is not a terribly intuitive interface (no link resolver that I’m aware of has one) and there are some items it just doesn’t resolve well, such as conference proceedings. Since conference proceedings and technical reports are huge for us, I decided we needed something that resolved these items better. That’s when the idea of the übeResolver (now mainly known as ‘the umlaut’) was born.

    Although I had been working with Ed Summers on the Ruby OpenURL libraries before Code4Lib 2006, I really began working on umlaut earlier this month when I thought I might have something coherent together in time before the ELUNA proposal submission deadline. Although I barely had anything functional on the 8th (the deadline — 2 days after I had really broken ground), I could see that this was actually feasible and doable.

    Three weeks later and it’s really starting to take shape (although it’s really, really slow right now). Here are some examples:

    The journal ‘Science’

    A book: ‘Advances in Communication Control Networks’

    Conference Proceeding

    Granted, the conference proceeding is less impressive as a result of IEEE being available via SFX (although, in this case, it’s getting the link from our catalog) and the fact that I’m having less luck with SPIE conferences (they’re being found, but I’m having some problems zeroing in on the correct volume — more on that in a bit), but I think that since this is the result of < 14 days of development time, it isn't a bad start. Now on to what it's doing. If the item is a "book", it queries our catalog for ISBN; asks xISBN for other matches, queries our catalog for that; does a title/author search; does a conferenceName/title/year search. If there are matches, it then asks the opac for holdings data. If the item is either not held or not available, it does the same to our consortial catalog. Currently it’s doing both, regardless, because I haven’t worried about performance.

    It checks the catalog via SRU and tries to fill out the OpenURL ContextObject with more information (such as publisher and place). This would be useful to then export into a citation manager (which most link resolvers have fairly minimal support for). While it has the MODS records, it also grabs LCSH and Table of Contents (if they exist). When I find an item with more data, I’ll grab it as well (such as abstracts, etc.).

    It then queries Amazon Web Services for more information (editorial content, similar items, etc.).

    It still needs to check SFX, but, unfortunately, that would slow it down even more.

    For journals, it checks SFX first. If there’s no volume, issue, date or article title, it will try to get coverage information. Unfortunately, SFX’s XML interface doesn’t send this, so I have to get this information from elsewhere. When I made our Ejournal Suggest service, I had to create a database of journals and journal titles and I have since been adding functionality to it (since I am running reports from SFX for titles and it includes the subject associations, I load them as well — it includes coverage, too, so including that field was trivial). So when I get the SFX result document back, I parse it for its services (getFullText, getDocumentDelivery, getCitedBy, etc.) and if no article information is sent, I make a web service request to a little PHP/JSON widget I have on the Ejournal Suggest database that gets back coverage, subjects and other similar journals based on the ISSN. The ‘other similar journals’ are 10 (arbitrary number) other journals that appear in the same subject headings, ordered by number of clickthroughs in the last month. This doesn’t appear if there is an article, because I haven’t decided if it’s useful in that case (plus the user has a link to the ‘journal level’ if they wish).

    Umlaut then asks the opac for holdings and tries to parse the holdings records to determine if a specific issue is held in print (this works well if you know the volume number — I have thought about how to parse just a year, but haven’t implemented it yet). If there are electronic holdings, it attempts to dedupe.

    There is still a lot more work to do with journals, although I hope to be able to implement this soon. The getCitedBy options will vary from grad students/faculty to undergrads. Since we have very limited seats to Web of Science, undergraduates will, instead, get their getCitedBy links to Google Scholar. Graduate students and faculty will get both Web of Science and Google Scholar. Also, if no fulltext results are found, it will then go out to the search engines to try to find something (whether it finds the original item or a postprint in arxiv.org or something). We will also have getAbstracts and getTOCs services enabled so the user can find other databases that might be useful or table of content services, accordingly. Further, I plan on associating the subject guides with SFX Subjects and LCC, so we can make recommendations from a specific subject guide (and actually promote the guide a bit) based, contextually, by what the user is already looking at. By including the SFX Target name in the subject items (which is an existing field that’s currently unused), we could also match on the items themselves.
    The real value in umlaut, however, will come in its unAPI interface. Since we’ll have Z39.88 ContextObjects, MODS records, Amazon Web Services results and who knows what else, umlaut could feed an Atom store (such as unalog) with a whole hell of a lot of data. This would totally up the ante of scholarly social bookmarking services (such as Connotea and Cite-U-Like) by behaving more like personal libraries that match on a wide variety of metadata, not just url or title. The associations that users make can also aid umlaut in recommendations of other items.

    The idea here is not a replacement of the current link resolver, the intention is to enhance it. SFX makes excellent middleware, but I think it’s interface leaves a bit to be desired. By utilizing its strength, we can layer more useful services on top of it. Also, a user can add other affiliations that they belong to in their profile, so umlaut can check their local public library or, if they are taking classes at another university, they can include those.

    At this point I can already hear you saying, “But Ross, not everyone uses SFX”. How true! I propose a microformat for link resolver results that could be parseable by umlaut (and in an ‘eating your own dog food’ fashion, will add this to umlaut’s template, eventually), making any link resolver available to umlaut.

    There is another problem that I’ve encountered while working on this project, though, too. Last week and the week before, while I was doing the bulk of the SRU development, I kept on noticing (and reporting) our catalog (and, more often, it’s Z39.50 server) going down. Like many times a day. After concluding that, in fact, I was probably causing the problem, I finally got around to doing something that I’ve been meaning to do for months (and I would recommend to everyone else if they want to actually make useful systems): exporting the bib database into something better. Last week I imported our catalog into Zebra and sometime this week I will have a system that syncs the database every other hour (we already have the plumbing for this for our consortial catalog). I am also experimenting with Cheshire3 (since I think it’s potential is greater — it’s possible we may use both for different purposes). The advantage to this (besides not crashing our catalog every half hour) is that I can index it any way want/need to as well as store the data any way I need to in order to make sure that users get the best experience they can.

    Going back to the SPIE conferences, there is no way in Voyager that I can limit my results to less than 360+ results for “SPIE Proceedings” in 2003. At least, not from the citations I get from Compendex (which is where anyone would get the idea to look for SPIE Proceedings in our catalog, anyway). With an exported database, however, I could index the volume and pinpoint the exact record in our catalog. Or, if that doesn’t scale (for instance, if they’re all done a little differently), I can pound the hell out our zebra (or cheshire3 or whatever) server looking for the proper volume without worrying about impacting all of our other services. I can also ‘game the system’ a bit and store bits in places that I can query when I need them. Certainly this makes umlaut (and other services) more difficult to share to other libraries (at least, other libraries that don’t have similar setups to ours), but I think these sorts of solutions are essential to improving access to our collections.

    Oh yeah, and lest you think that mirroring your bib database is too much to maintain: Zebra can import marc records (so you can use your opac’s marc export utility) and our entire bib database (705,000 records) takes up less than 2GB of storage. The more indexes added, the larger the database size, of course, but I am indexing a LOT in that.

  • Beware, Birds of Valhalla: Fern [1993-2/26/2006], R.I.P.

    Sorry to say, Mr. Eliot, but you’re an idiot. February is the cruelest month, by far.

    Fern was my first dependent, my college cat. I got her for my twenty-first birthday from Knoxville’s Adopt-a-pet. The index card taped to the wall read:

    name: Candace

    quote: “I don’t like these other cats. Take me home!”

    And I did take her home (and changed her name). She was a beautiful cat with a lot of personality (and Candace was an entirely improper name for her). Gray with black stripes and spots, two “rust spots” around her neck, and small black dot on her lip. She quickly took over my life.

    Fern was not a good cat. In fact, she was downright evil. She demanded undivided attention, and when she didn’t get it, bad things would happen. I often speculated that if she had an opposable digit, we’d be screwed. Certainly, she left an impression. Especially on our Rottweiler, Ava.

    Ava loved her kitty. She would follow her all over the house in a very “Spy vs. Spy” manner (earning Fern the nickname “Secret Agent Kitty”). The kitty just confounded her and when Fern fell ill and we had to confine her to the bathroom, Ava just sat outside the bathroom door for a month, staring at the door, waiting for her.

    She left a large impression on my love life, as well. She gave Selena cat-scratch fever. She would always sit in between me and any girl I was dating on my couch. She would rattle the hell out of my bedroom door at 6:00 in the morning. Her presence inspired me to write a song about her (“All the women I’ve ever loved have been allergic to my cat”). She was good for things like that.

    Most would remember Fern as the cat that lived in a wine crate over the refrigerator, scaring the unsuspecting person going for a beer. Others might remember her for being the cat the brought home a bird in the middle of a party I was having.
    Fern died last night at around 7PM of renal failure (she had been battling kidney problems for over a year) She was in my lap and purring (oh how she would purr). When we laid her out in her wine crate for the dogs to see, Ava licked her nose.

    We buried her in the backyard in the Hello Kitty bed my mother made for her for Christmas.

    Fern, may be the birds be plentiful.

  • Safe and sound at home

    I had to leave Code4lib 2006 early, and to all there that were concerned — things are ok here (but it’s good I came home).

    I cannot believe how well the conference went and having read some posts about day three, it seems like it ended great.

    I’m already looking forward to next year.

  • Opa (y’all)!

    Code4lib 2006 day 1 went exceedingly well.  Much better than I had expected (only because I am one of the “organizers”), in fact.

    I was nervous about starting off with a “virtual keynote” (PINES, literally, just phoned in their presentation), especially since I was the goob that had to advance their PPT when they made a “beeping” noise.  It went great, though!

    The only “problem” (which actually has a hidden “advantage”) is that we’re cramming so much stuff in such a short amount of time, the scheduling and timekeeping gets to be a bit draconian.  On the flipside, the energy and momentum is so high, and so much is being said, that it certainly makes up for having to occasionally rush people/cut them off completely.

    The unAPI breakout session was remarkably productive.  I think we’re actually going to have a useful spec soon!

    My lightning talk didn’t go so well, but I’m kicking around the idea of another one tomorrow, so maybe that will balance things.

    I’m really happy with where this is going.  It’s great to meet these like minds face to face and I think a lot of ideas (and energy) are brewing.

    Speaking of brewing — time to go drink some local beer.

    OPA!