Dilettante's Ball

Category: libraries

  • Explosive Diaereses

    I’ve been waiting for a while to have this title. Well, actually, not a long while, and that’s testimony to how quickly I’m able to develop things in Rails.

    While I think SFX is fine product and we are completely and utterly dependent upon it for many things, it does still have its shortcomings. It is not a terribly intuitive interface (no link resolver that I’m aware of has one) and there are some items it just doesn’t resolve well, such as conference proceedings. Since conference proceedings and technical reports are huge for us, I decided we needed something that resolved these items better. That’s when the idea of the übeResolver (now mainly known as ‘the umlaut’) was born.

    Although I had been working with Ed Summers on the Ruby OpenURL libraries before Code4Lib 2006, I really began working on umlaut earlier this month when I thought I might have something coherent together in time before the ELUNA proposal submission deadline. Although I barely had anything functional on the 8th (the deadline — 2 days after I had really broken ground), I could see that this was actually feasible and doable.

    Three weeks later and it’s really starting to take shape (although it’s really, really slow right now). Here are some examples:

    The journal ‘Science’

    A book: ‘Advances in Communication Control Networks’

    Conference Proceeding

    Granted, the conference proceeding is less impressive as a result of IEEE being available via SFX (although, in this case, it’s getting the link from our catalog) and the fact that I’m having less luck with SPIE conferences (they’re being found, but I’m having some problems zeroing in on the correct volume — more on that in a bit), but I think that since this is the result of < 14 days of development time, it isn't a bad start. Now on to what it's doing. If the item is a "book", it queries our catalog for ISBN; asks xISBN for other matches, queries our catalog for that; does a title/author search; does a conferenceName/title/year search. If there are matches, it then asks the opac for holdings data. If the item is either not held or not available, it does the same to our consortial catalog. Currently it’s doing both, regardless, because I haven’t worried about performance.

    It checks the catalog via SRU and tries to fill out the OpenURL ContextObject with more information (such as publisher and place). This would be useful to then export into a citation manager (which most link resolvers have fairly minimal support for). While it has the MODS records, it also grabs LCSH and Table of Contents (if they exist). When I find an item with more data, I’ll grab it as well (such as abstracts, etc.).

    It then queries Amazon Web Services for more information (editorial content, similar items, etc.).

    It still needs to check SFX, but, unfortunately, that would slow it down even more.

    For journals, it checks SFX first. If there’s no volume, issue, date or article title, it will try to get coverage information. Unfortunately, SFX’s XML interface doesn’t send this, so I have to get this information from elsewhere. When I made our Ejournal Suggest service, I had to create a database of journals and journal titles and I have since been adding functionality to it (since I am running reports from SFX for titles and it includes the subject associations, I load them as well — it includes coverage, too, so including that field was trivial). So when I get the SFX result document back, I parse it for its services (getFullText, getDocumentDelivery, getCitedBy, etc.) and if no article information is sent, I make a web service request to a little PHP/JSON widget I have on the Ejournal Suggest database that gets back coverage, subjects and other similar journals based on the ISSN. The ‘other similar journals’ are 10 (arbitrary number) other journals that appear in the same subject headings, ordered by number of clickthroughs in the last month. This doesn’t appear if there is an article, because I haven’t decided if it’s useful in that case (plus the user has a link to the ‘journal level’ if they wish).

    Umlaut then asks the opac for holdings and tries to parse the holdings records to determine if a specific issue is held in print (this works well if you know the volume number — I have thought about how to parse just a year, but haven’t implemented it yet). If there are electronic holdings, it attempts to dedupe.

    There is still a lot more work to do with journals, although I hope to be able to implement this soon. The getCitedBy options will vary from grad students/faculty to undergrads. Since we have very limited seats to Web of Science, undergraduates will, instead, get their getCitedBy links to Google Scholar. Graduate students and faculty will get both Web of Science and Google Scholar. Also, if no fulltext results are found, it will then go out to the search engines to try to find something (whether it finds the original item or a postprint in arxiv.org or something). We will also have getAbstracts and getTOCs services enabled so the user can find other databases that might be useful or table of content services, accordingly. Further, I plan on associating the subject guides with SFX Subjects and LCC, so we can make recommendations from a specific subject guide (and actually promote the guide a bit) based, contextually, by what the user is already looking at. By including the SFX Target name in the subject items (which is an existing field that’s currently unused), we could also match on the items themselves.
    The real value in umlaut, however, will come in its unAPI interface. Since we’ll have Z39.88 ContextObjects, MODS records, Amazon Web Services results and who knows what else, umlaut could feed an Atom store (such as unalog) with a whole hell of a lot of data. This would totally up the ante of scholarly social bookmarking services (such as Connotea and Cite-U-Like) by behaving more like personal libraries that match on a wide variety of metadata, not just url or title. The associations that users make can also aid umlaut in recommendations of other items.

    The idea here is not a replacement of the current link resolver, the intention is to enhance it. SFX makes excellent middleware, but I think it’s interface leaves a bit to be desired. By utilizing its strength, we can layer more useful services on top of it. Also, a user can add other affiliations that they belong to in their profile, so umlaut can check their local public library or, if they are taking classes at another university, they can include those.

    At this point I can already hear you saying, “But Ross, not everyone uses SFX”. How true! I propose a microformat for link resolver results that could be parseable by umlaut (and in an ‘eating your own dog food’ fashion, will add this to umlaut’s template, eventually), making any link resolver available to umlaut.

    There is another problem that I’ve encountered while working on this project, though, too. Last week and the week before, while I was doing the bulk of the SRU development, I kept on noticing (and reporting) our catalog (and, more often, it’s Z39.50 server) going down. Like many times a day. After concluding that, in fact, I was probably causing the problem, I finally got around to doing something that I’ve been meaning to do for months (and I would recommend to everyone else if they want to actually make useful systems): exporting the bib database into something better. Last week I imported our catalog into Zebra and sometime this week I will have a system that syncs the database every other hour (we already have the plumbing for this for our consortial catalog). I am also experimenting with Cheshire3 (since I think it’s potential is greater — it’s possible we may use both for different purposes). The advantage to this (besides not crashing our catalog every half hour) is that I can index it any way want/need to as well as store the data any way I need to in order to make sure that users get the best experience they can.

    Going back to the SPIE conferences, there is no way in Voyager that I can limit my results to less than 360+ results for “SPIE Proceedings” in 2003. At least, not from the citations I get from Compendex (which is where anyone would get the idea to look for SPIE Proceedings in our catalog, anyway). With an exported database, however, I could index the volume and pinpoint the exact record in our catalog. Or, if that doesn’t scale (for instance, if they’re all done a little differently), I can pound the hell out our zebra (or cheshire3 or whatever) server looking for the proper volume without worrying about impacting all of our other services. I can also ‘game the system’ a bit and store bits in places that I can query when I need them. Certainly this makes umlaut (and other services) more difficult to share to other libraries (at least, other libraries that don’t have similar setups to ours), but I think these sorts of solutions are essential to improving access to our collections.

    Oh yeah, and lest you think that mirroring your bib database is too much to maintain: Zebra can import marc records (so you can use your opac’s marc export utility) and our entire bib database (705,000 records) takes up less than 2GB of storage. The more indexes added, the larger the database size, of course, but I am indexing a LOT in that.

  • The Librarian/Staff Divide

    Every library I have worked at has had an uneasy caste system between the faculty and staff. While I understand this to extent, this delineation is used without rhyme or reason much of the time. The implication is that this means the librarians are treated as “career professionals” and the staff is merely “the help” (more on this in a minute).

    I was pleasantly surprised to see the University of Iowa waive the MLS requirement for their current “Director of Information Technology” job posting (MS Word Document).

    The University of Iowa Libraries seeks a creative, experienced professional to lead our information technology (IT) operations. Building on the Libraries’ current capabilities, the Director will provide innovative leadership in the use of technology to deliver information resources and services to the Libraries’ user communities. The Director for Library Information Technology reports directly to the University Librarian and is a member of the Libraries’ Executive Council contributing to overall strategic planning, program development and evaluation, and the allocation of resources in support of the Libraries’ mission.

    This is a senior administrative position responsible for IT planning, the development of system-wide policies and procedures, and the coordination of information technology activities throughout the library system. The Director will supervise a department of 11 staff responsible for desktop support and technical training, systems administration and security/rights management, and applications development. Collaborative and advocacy activities with other library administrators and staff as well as members of the IT community on campus, the state, nationally and internationally are key responsibilities of this position. This Director serves as the Libraries’ liaison to CNI, EDUCAUSE, and similar organizations.

    Qualifications

    Required:

    • Bachelor’s degree

    • Minimum of 9 years of library information technology experience in a university environment

    • Demonstrated knowledge of current trends and best practices in the application of information technology in research libraries and higher education

    • Demonstrated experience promoting and working effectively in a diverse environment

    • Evidence of highly effective interpersonal and communication skills.

    • Evidence of analytical and creative problem-solving skills

    • Library-wide perspective and ability to contribute to planning and system-wide administration of the Libraries

    • Record of active participation in national pertinent professional associations

    While I meet the “minimum requirements” for this position, I am not fully qualified… somewhere in this job’s requirements (maybe not actually written in it) is a healthy desire to live in or near Iowa City (which I do not possess). Still, this is very, very progressive for school of UIowa’s size. They are indicating that if you have spent 9 years of your life working in libraries and have shown the initiative to participate professionally and whatnot, you’re the sort of person who they want on their team. The idea being that the MLS is not a terribly good indicator of skills and (for this sort of position, anyway) may actually limit the pool of good potential candidates they may get.
    When I was hired as “Web and Application Development Coordinator” at Emory, they stripped any “faculty-ness” from the position which, in turn, devalued the authority the position had — at least this seemed true in practice. The position could have been pretty similar to that of Iowa’s (I still would have been miserable at it), but the faculty wasn’t quite able to make the leap to label me a “peer”.

    Just like my Emory position, though, Iowa is requiring that the position be a “manager” position to get invited to the “career professional” table. There are a lot of librarians at that table who are not managers.

    This gap was made evident eariler this week here at Tech. I frequently hear about things going on in the library secondhand due to the fact that I am not faculty. Lord knows how much I never hear. We have a mailing list (lib-fac) where “career professional” sorts of announcements are made and, on Tuesday, a meeting was held for the faculty only to hear people report back what they had learned at ALA Midwinter.

    My beef here is, why the faculty only distinction? If they were talking about tenure review or the sorts of things that couldn’t affect me by nature of my employment status that would be one thing. They are talking about things that I would like to know about, however, and could possibly contribute a voice in the discussion. I am not entirely sure why I should be left out.

    It’s possible that I could be included just by asking to be. But why am I (or any other staff that has an interest in the profession, for that matter) being excluded in the first place?

    I guess what I’m trying to say is that it’s a little discouraging to work to be treated as a peer and a colleague in a national/international community and not in my own organization.

    My intention here is not to single out Georgia Tech; this is prevalent throughout libraries everywhere (at least, from what I’ve seen it has). I’m just calling for the possibility of a third caste: those that are making a career in libraries, but have no desire for faculty status (or management).

  • That Article I Had No Business Writing…

    Is finally out.
    So, uh, if you’re in the market to buy a Link Resolver or just are seeking general information on one… er, check it out.

  • To Free or not to Free?

    Last week I was reading Dorothea Salo’s posting about OCLC’s report on library branding, and it got me to thinking about this a bit.

    In particular, I thought about her comment:

    I would want to trial-balloon a “Deep Web” play in my next survey, if I were OCLC. I would want to know how many people have heard of the Deep Web, what they think is in it, whether they think information useful to them is in it, whether they would access it through their libraries if they could. This moves away from free-vs.-paid and toward exclusive-vs.-nonexclusive. People like the idea of being privileged. If the library is a place that privileges them, I think they’ll go for it. Special-collections and archives get a boost in this campaign, too; access to rare or unique information is the ultimate in privilege.

    I see a tension here. While many push for an “information wants to be free” model, this would, inherently, devalue the role of the organization that makes it free. In fact, to take her quote even farther, this is especially true of special collections and archives.

    Allow me to explain.

    Users aren’t particularly discriminatory as to where they get their information. Our students or faculty don’t really care if the article or research they are looking at comes to them courtesy of Georgia Tech or if it was found in Citeseer. They are more likely to say they found something in “Google Scholar” vs. the actual institutional repository for the school they are actually getting it from. The more open the information is, the less exclusive our collection becomes and the less leverage and value we hold (at least conforming to our traditional model).

    With special collections, this is especially true. Special collections are “special” because they are “unique”. Libraries spend a lot of money curating these collections. Historically, this has enjoyed a fairly good ROI because it distinguishes the library (and therefore, larger institution) as something “special” itself. These materials are exclusive to that particular institution and give value to the collection.

    However, there is pressure to digitize and publish these collections. If all of these collections are digitized and published, we have a bunch of silos strewn about the internet requiring the user know about find them to use them. Since it is a lot of work to digitize and mark up these collections, there’s not a terribly good return for the effort.

    In an effort to improve findability, the collections need to be aggregated with other similar collections to increase their exposure. However, the result of this is improved awareness and accessibility, but at the same time it dilutes exclusiveness and branding. Whoever provides the aggregation/discovery service gets the benefit of the content, so some of the content providers (inherently) must lose.

    So, what does this mean? It should not prevent us from making our collections more open and accessible. That runs counter to our mission. However, we need to start thinking of ways to generate value when our information is free. There are plenty of ways of doing that, such as tailoring services that aggregates the “free” information for our communities, or building systems that can use the information in unique and specialized ways.

    There is a large cultural shift that needs to take place to realize this future, however. We still place a lot of emphasis (way too much, really) on the size and uniqueness of our collections. With a world of information available (or a lot of it, at any rate), it’s not so much an issue of how many books you have in your building, but how you are able harness all the good data and present it in useful and meaningful ways. There aren’t easy metrics to this. ARL just can’t count book spines and annual budget. Serious consideration needs to be paid to what and how a library is utilizing the collection outside their walls.

  • Rails Resolver Router: on Rails!

    Since my foray into python a couple of months ago, I’ve been enjoying branching out into new languages.

    I had pitched the concept of a link resolver router for the state universal catalog to a committee I sit on (this group talks about SFX links in the 856 tag and whatnot). The problem with making links for a publicly available resource point to your institutional resolver is just that. It’s pointing your your institutional resolver, despite the fact that your audience could be coming from anywhere. This plays out even greater in a venue such as a universal catalog, since there’s not really a “home institution” to point a resolver link, anyway. OCLC and UKOLN both have resolver routers, and OCLC’s certainly is an option, but I don’t feel comfortable with the possibility that all of our member institutions might have to pay for the service (in the future). My other problem with OCLC’s service is that you can only belong to one institution and I have never liked that (especially as more and more institutions have link resolvers).

    So, in this committee I mentioned that it would be pretty simple to make a router, and since I was having trouble getting people to understand what exactly I was talking about, I decided to make a proof-of-concept. And, since I was making a proof-of-concept, I thought it’d be fun to try it in Ruby on Rails.

    Now, a resolver router is about the simplest concept possible. It doesn’t really do anything but take requests and pass them off to the appropriate resolver. It’s a resolver abstraction layer, if you will. I thought this was a nice, small project to try to cut my Ruby teeth on. There’s a little bit a database, a little bit of AJAX. It’s also useful, unlike making a cookbook from a tutorial or something.

    It took about three days to make this. After you pick your resolver (Choose a couple! Add your own!), you’ll be taken to a page to choose between your various communities for appropriate copy.

    I chose this particular citation because it shows the very huge limitation of link resolvers (if you choose Georgia Tech’s resolver and Emory’s resolver, for instance); despite the fact that this is freely available, it does not appear in my resolver. That’s not really the use case I envision, though. I am thinking more of a case like my co-worker, Heather, who should have access to Georgia Tech’s collection, Florida State’s resources (she’s in grad school there), Richland County Public Library (she lives in Columbia, SC), and the University of South Carolina (where her husband is a librarian). The resolver router alleviates the need to search for a given citation in the various communities (indeed to even have to think of or know where to look within those communities).

    Sometime later this winter, I’ll have an even better use case. I’ll keep that under wraps for now.

    Now, my impression of Ruby on Rails… For a project like this, it is absolutely amazing. I cannot believe I was able to learn the language from scratch and implement something that works (with this amount of functionality) in such a short amount of time. By bypassing the need to create the “framework” for the application, you can just dive into implementation.

    In fact, I think my time to implementation would have been even faster if the number of resources/tutorials out there didn’t suck out loud. Most references point to these tutorials to get you started, but they really aren’t terribly helpful. They explain nothing about why they are doing what they are doing in them. I found this blog posting to be infinitely more useful. Her blog in general is going in my aggregator, I think.

    When it comes to learning Ruby, this is a masterful work of art… but… not terribly useful if you just want to look things up. I recommend this for that.

    Anyway, I am so impressed with Ruby on Rails that I am planning on using it (currently) for “alternative opac project“, which is now being code named “Communicat”. More on this shortly (although I did actually develop the database schema today).

  • Envisioning Alto

    I have mentioned here several times the “alternative to the catalog” project I am trying to implement at Tech. One of the problems that I’ve had is naming the project something that lets people realize what I’m talking about, without the political hairiness of saying “catalog replacement” (since that’s technically not true, anyway).

    In a meeting two weeks ago (about subject guides), I was drawing the concept of this project on the whiteboard of our conference room. It’s been up ever since and in the middle, I had written “ALTOpac” because that was an easy way to loosely describe it in a way that the uninitiated in the room could envision where I was starting from. Sitting in another meeting today, the capitalized letters jumped out at me: ALTO. It means nothing.

    And I like that. Of course it still doesn’t explain what it’s about. That’s what subtitles are for.

    Now, let me explain what the hell Alto is and what it is supposed to do.

    Alto is a “community-based collection builder and search engine”.

    Come to think of it, that might not actually clear anything up.

    Let’s back up a bit, shall we?

    To say searching the catalog is “searching our collection” is quite arbitrary and false. Metasearch doesn’t really solve this problem, since you’d still only point the metasearch engine at certain assets and it’s non-trivial to make relationships between assets. Metasearch is part of the solution, but hardly the panacea.

    Again, our “collection” is an ambiguous term and shouldn’t be solely determined by our collection development policies/budget. It is our opinion that if something is important enough to be added to a reserves list (even a web page), it should technically be part of our collection. I would not, however, say it should be cataloged (and that’s why this isn’t a catalog replacement project, see?). If an item is even bookmarked (via a local social bookmarking service, such as unalog or connotea) it should then become part of our collection. A 1927 engineering textbook from Purdue’s catalog? Index it! If a member of our community finds it important enough to want to come back to and share with a group, it’s important enough for us to aggregate into our “collection”. Relevance comes later (keep reading, if you’re interested).

    There are also relationships that our community (for the sake of argument, let’s start with “Georgia Tech”) builds that are highly relevant for finding connections between disparate “things”. So, the items put on reserve for a particular course have an umbrella of commonality between them that should be utilized for anyone that runs across any of these items. The relevance ranking should be even greater for a user that happens to be a member of the group in question (for instance, is enrolled in the class).

    If Alto has a citation management-esque feature in it, users can very specifically group relevant resources together based on a project. Resources can be anything: books, websites, articles, searches, chat transcripts, trails, you name it.

    And all of this should feed the “relevance beast”, as it were.

    So that’s some background. Given that we’ll have some formal subject classifications for these objects (from the OPAC or from metasearch or whatever), we should be able to bridge the formal to folksonomy to make sense of how people have classified their saved things.

    We can then begin to cluster search results. Format, subject, concept, group, policies… All of these can be browsed after the search begins. The search results will be a combination of metadata objects and library content. If some of the results appear in a given “subject guide”, the guide will a suggested resource (and will, in turn, push some resources into the result set).

    The goal is to open the silos we have created around our resources/services. It would break down the ambiguity between “collections”, “services” and “policies” since they’re all interrelated.

    How do we plan to do this? Glad you asked (you’re still reading, right?)!

    We’ve exported all of the bib records from our catalog. The plan is to use METS as our wrapper around MODS. We’ll then harvest our institutional repository and index our website. That’s a pretty good base to start with. All of this is stored in a dbXML database and indexed with Lucene.

    If users want to harvest a collection from citeseer or OAIster, that will be available and will become part of our collection. Annotations, links to reviews, links to content to index will all be made available.

    I’m leaving a lot out and glossing some of this over… but it starts to put the idea on “paper” for me to come back later.

  • I sound my barbaric YAWP over the walls of my cubicle.

    I woke up at 4:30 this morning.

    One could easily write this off to a variety of stresses: an article I have no business writing; a conference I have no business helping organize; a huge project that I am having problems getting started on; a house that I apparently haven’t sunk enough money in to move into yet; a house that I can’t drag far enough away from the railroad tracks to sell; the usual burden that is “the holidays”… sure one could try to pin it on any of those.

    But I woke up thinking about (meaning that I was dreaming about) something I read recently from Richard Wallis on Panlibus, Talis’ ‘blog:

    Well yes, the current generation of ILS systems were not built with Web Services everywhere. To put it bluntly, who will pay the salaries of the developers who are going to develop these services for you to consume?

    Strange thing to dream about, I know. However, when I think about this one quote, it pisses me off to no end. The University System of Georgia pays Endeavor over $500,000 a year for the privilege of running an ILS that they haven’t invested any innovation in years. Granted, we are 35 libraries, so it’s not like we’re all paying that ransom, but, on the flip side, we’re probably also getting a discount for the very fact that we are so large.

    Then, to think we are but a percentage of Endeavor’s total customer base…

    WHERE IS THAT MONEY GOING, RICHARD?

    Of course, I realize that Talis is in no way related to Endeavor, but I cannot imagine their pricing is so radically different that their coffers have no shillings to pay for developers.

    Besides, they must already have developers, right? Maybe you need hire developers with vision.

    So, to this argument, I call bullshit.

    The other thing that struck me (again, apparently in my dream) is the apologetic tone I see quite frequently (recent example here, lots of others floating about) that shifts the blame of our stagnant, crappy Integrated Library Systems to us, the customers instead to our vendors. The argument goes that we, the libraries, have asked for the wrong things for the ILS and the poor vendors (poor, poor vendors) had their hands tied, literally tied, trying to keep up with our demands to be able to incorporate any sort of innovation in the last 15-20 years. Besides, they’d say, if they came up with something different, libraries might not want the change.

    What (successful) technology company has ever relied on RFPs for their innovation? Are Google’s hands tied until some customer says, “Hey, can you make a web based ‘maps’ site? You know what we need? A new way to do threaded email.”? How about Intel? Microsoft?

    No. These companies realize that they need to innovate to survive. To stagnate or half-ass is the kiss of death. See Novell. For a more dramatic example, see Apple.

    No, it’s time we stop taking it like abused spouses from our vendors. You know, maybe we did overcook the porkchop and maybe we do open our mouths too much, but that’s no reason to have a black eye. If a handful of the better funded libraries were to help found something like the Apache Foundation for library software, our abusive husbands might find treat as partners rather than punching bags. I think I might know a good place to look for talent.

    (In truth, our rottweiler woke me up, but the dream still stands).

  • Library 1.7.02-4 pre 6

    I really, really hate this Library 2.0 meme for a couple of reasons.

    1) All of our problems will not, in fact, be solved with AJAX and web interfaces

    2) In fact many of our problems cannot be solved by technology at all (try doing interesting and meaningful and different work with the current body of MARC records out there and see what I mean)

    3) This quest for 2.0 would be better served if “2.0” was a milestone on the journey to “Library 4.5” — I mean, come on folks, let’s get back into innovating.

    4) I think it trivializes some actually exciting and useful work that I fear will continue to fly under the radar because it’s not “Web 2.0” enough.

    Maybe hype is necessary to rally the troops, but I really wish vision would get more attention.

  • Reactions from Access 2005

    I’ll have to keep this rather short, since the hotel wireless network is being flaky (as usual) (in fact I am having to write this while standing in the bathroom — oddly the best wireless reception in my room).

    Again, the conference organizers have proven why this is the only professional event I schedule in my year. I’ll comment on the earlier days a little later, but while day 3 is still rather fresh in my mind (as fresh as my poor, oversaturated mind can be), I’d like to touch on a few things.

    1) Listen to every word Lorcan says, always. It blows my mind what an amazing asset he is to our community and how there are people (in my library) who have no idea who he is. His presentation this morning has completely energized me to kick it up a notch in getting our library more into our (and other) user’s “LifeFlow”.

    2) Art and Peter proved that WAG needs no “G”. I’ve been working with Art on this WebDav/OPAC project for months (thanks to SUDOC, as he pointed out), and I never, ever would have dreamed of the things these two are coming up with. Cocoon is, evidentally, a very magical beast and the potential of storing these “trails” could have huge implications on the collaborative research environment that we are trying to create at Georgia Tech. Being able to chart the path of scholarship would make it easier to get to the giant to stand upon his shoulders.

    3) Internet communication is lousy when trying to develop a new spec. Despite being there at the beginning (and being a very loud proponent of COinS), I could not wrap my head around the use cases for COinS-PMH. Oh, Dan tried to “learn me”, but it really took his presentation today to “get it”. I definitely “get it” now, and expect to see COinS-PMH all over Tech.

    4) Hackfest is the greatest invention ever. And I honestly couldn’t imagine it working properly at any other conference (sorry, LITA).

    5) This is why I’m applying to enroll in the Master’s program for Human-Computer Interaction at Tech. This, coupled with the previous two presentations (Art/Peter’s, Dan’s)… Holy crap. The world would be so different.

    6)     The U.S. is screwed. We have sold our souls, culture and future to corporate interests and I’m not sure how we can fix it. As Peter remarked to me, hopefully Cliff Lynch’s vision of a world where everything is digitized except the intellectual output after 1920 will light a fire under us. I fear at that point it may be too late, however. It looks like Canada’s future might be a bit brighter. Even if it isn’t, I’ll get fired up by the revolutionary rhetoric, any day.

    Wow, I love this place.

  • But, you see, my wheel is nothing like all of those other seemingly identical wheels

    I am still feeling my way around Python. I have yet to grasp the zen of being Pythonic, but I am at least coming to grips with real object orientation (as opposed to the named hashes of PHP) and am actually taking the leap into error handling, which, if you have dealt with any of the myriad bugs in any of my other projects, you’d know has been a bit of a foreign concept to me.

    Python project #2 is known as RepoMan (thanks to Ed Summers for the name). It attempts to solve a problem that not one but two other opensource projects already have solved admirably (I’ll go into more about this in a bit). RepoMan is an OAI Repository indexer that makes said repository available via SRU. I created it in an attempt to make our DSpace implementation searchable from remote applications (namely, the site search and the upcoming alternative opac). It’s an extremely simple two script project that has only taken a week to get running largely due to the existence of two similar and available python scripts that I could modify for my own use. It’s also due to the help of Ed Summers and Aaron Lav.

    The harvester is, basically, Thom Hickey’s one page OAI harvester with some minor modification. I have added error handling (the two lines I added to compensate for malformed xml must have been over the “one page limit”) and instead of outputting to a text file, it shoves the records in a Lucene index (thanks to PyLucene). This part still needs some work (I’m not sure what it would do with an “updated” record, for example), but it makes a nice index of the Dublin Core fields, plus a field for the whole record, for “default” searches. This was a good exercise for me to work with xml, Python and Lucene, because I was having some trouble when trying to index the MODS records for the alternative opac.

    The SRU server is, basically, Dan Chudnov‘s SRU implementation for unalog. It needed to be de-Quixotefied and is, in fact, much more robust than Dan’s original (of course, unalog’s implementation doesn’t need to be as “robust”, since the metadata is much more uniform), but certainly having a working model to modify made this go much, much faster. The nice part is that there might be some stuff in there that Dan might want to put back into unalog.

    So, here is the result. The operations currently supported are explain and searchRetrieve and majority of CQL relations are unsupported, but it does most of the queries I need it to do and, most importantly, it’s less than a week old.

    So the burning question here is: why on earth would I waste time developing this when OCKHAM’s Harvest-to-Query is out there, and, even more specifically, OCLC’s SRW/U implementation for DSpace is available? Further, I knew full well that these projects existed before I started.

    Lemme tell ya.

    Harvest-to-Query looked very promising. I began down this road, but stopped about halfway down the installation document. Granted, anything that uses Perl, TCL and PHP has to be, well, something… After all, those were the first three languages I learned (and in the same order!). Adding in IndexData’s Zebra seemed logical as well since it has a built-in Z39.50 server. Still, this didn’t exactly solve my problem. I’d have to install yazproxy, as well, in order to achieve my SRU requirement. Requiring Perl, TCL, PHP, Zebra and yazproxy is a bit much to maintain for this project. Too many dependencies and I am too easily distracted.

    OCLC’s SRW/U seemed so obvious. It seemed easy. It seemed perfect. Except our DSpace admin couldn’t get it to work. Oh, I inquired. I nagged. I pestered. That still didn’t make it work. I have very limited permissions on the machine that DSpace runs on (and no permissions for Tomcat), so there was little I could do to help. This also solved a specific purpose, but didn’t necessarily address any other OAI providers that we might have.

    So, enter RepoMan. Another wheel that closely resembles all the other wheels out there, but possibly with minor cosmetic changes. Let a thousand wheels be invented.