Dilettante's Ball

Blog

  • Well, that was strange

    I was watching one of the Rocketbooms my TiVo recorded for me, when I noticed that Joanne Colan “Twitters” with Ed Vielmetti.

    Apparently Superpatron‘s gone Hollywood.

  • In search of… Bigfoot

    Before I left for Guatemala, Ian Davis at Talis asked if I could give him a dump of our MARC records to load into Talis Platform. I had been talking in the #code4lib channel about how I was pushing the idea of using Talis Source to make simple, ad-hoc union catalogs; we could make one for Georgia Tech & Emory (we have joint degree programs) or Arche or Georgia Tech/Atlanta-Fulton Public Library, etc. My thinking was that by utilizing the Talis Platform, we could forgo much of the headache in actually making a union catalog for somewhat marginal use cases (the public library one notwithstanding).

    About a week after I got back from Guatemala, I had an email from Richard Wallis with some urls to play around with to access my Bigfoot store. He showed me search services, facet services and augment services. I was unable to be really dive into it much at the time but since I’m working on a total site search project for the library, I thought this would be a good chance to kick the tires a bit to include catalog results.

    After two days of poking around, I have made some opinions of it, have some recommendations for it, and wrote a Ruby library to access it.

    1) The Item Service

    This is certainly the most straightforward and for many people, the most useful service of the bunch. The easiest way to think of the item service is an HTTP based Lucene service (a la Solr or Lucene-WS) of your bib records. It returns something OpenSearch-y (it claims to be a RSS 1.0 document), but it doesn’t validate. That being said, FeedTools happily consumed it (more on that later) and the semantics should be familiar to anyone that has looked at OpenSearch before. Each item node also contains a Dublin Core representation of the record and a link to a marcxml representation. I’m not sure if there’s a description document for Bigfoot.
    Although the query syntax is pure Lucene (title:”The Lexus and the Olive Tree”), the downside is that it’s not documented anywhere what the indexes are and I doubt there would be any way to add new ones (for example, my guess is I wouldn’t be able to get an index for 490/440$v that I use for the Umlaut). I don’t see returning the results as OAI_DC being too much of a problem, since the RSS item includes a title (which would have been tricky between the DC and the marcxml). My Ruby library might not generate valid DC, I haven’t really looked into it.

    The docs also mention you can POST items to your Bigfoot store, but they don’t mention what your data needs to look like (MARC?) or what credentials you need to add something (I mean, it must be more than just your store name, right?). My hope is to add this functionality to bigfoot-ruby soon (especially since my data is from a bulk export from last October).

    2) The Facet Service

    This one is intriguing, definitely, since Faceted searching is all the rage right now.  The search syntax is basically the same as the Item Service, except you also send a comma delimited list of the fields you would like to query.  What you get back is either an XML or XHTML document of your results.

    For each field you request, you get back a set of terms (you can specify how many you want, with a default of 5) that appear most frequently in your field.  You also get an approximation for how many results you would get in that facet and a url to search on that facet.  It’s quite fast, although, realistically, you can’t do much with the output of facet search alone.

    Again, it’s difficult to know what you can facet on (subject, creator and date are all useful — I’m sure there are others) and the facet that (for me, at least) held the most promise — type — is too overly broad to do much with (it uses Leader position 7, but lumps the BKS and SER types all in a label called “text”).  I would like to see Talis implement something like my MARC::TypedRecord concept so one could facet on things like government document or conference.  You could separate newspapers from journals and globes from maps.  Still, the text analysis of the non-fixed fields is powerful and useful and beats the hell out of trying to implement something like that locally.

    In bigfoot-ruby, I have provided two ways to do a faceted search:  you can just do the search and get back Facet objects containing the terms and search urls or you can facet with items which executes the item searches automatically (in turn getting a definitive number of results for the query, as well).  Since I didn’t bother to implement threading, getting facets with items can be pretty slow.

    3) The Augment Service

    To be honest, I’m having a hard time figuring out useful scenarios for the augment service.  The idea is that you give it the URI of an RSS feed, and this service will enhance it with data from your Bigfoot store (at least, that is sort of how I understand it works).  Richard’s example for me was to feed it the output of an xISBN query (which isn’t in RSS 1.0, AFAIK, but, for the sake of example…) and the augment service would fill in the data for ISBNs your library holds.  The API example page mentions Wikipedia, but I don’t know where other than the Talis Platform that you can get Wikipedia entries formatted properly.  I tried sending it the results of an Umlaut2 OpenSearch query, but it didn’t do anything with it.  Presumably this RSS 1.0 feed needs the bib data to be sent in a certain way (my guess is in OAI_DC, like the Item Service), but I’m not sure.  The only use case I can think of for this service is a much simpler way to check for ISBN concordance (rather than isbn:(123456789X|223456789X|323456789X|etc.))

    Overall, I’m really impressed with the Talis API.  It is a LOT easier to use than, say, Z39.50 and by using OpenSearch seems more natural to integrate into existing web services than SRU.

    Bigfoot-ruby is definitely a work in progress.  I think I would like to split the Search class into ItemService and FacetService.  I don’t like how results is an Array for items and a Hash for facets. Just seems sloppy.  I need to document it, of course and I would like to implement Item POST.  This project also made me realize how bloody slow FeedTools is.  I am currently using it in both the Umlaut and the Finding Aids to provide OpenSearch, but I think it’s really too sluggish to justify itself.

    Thanks, Talis, for getting me started with Bigfoot and giving me the opportunity to play around with it.  Also, thanks to Ed Summers for fixing SVN on Code4lib.org.  You wouldn’t be able to download it and futz around with it yourself, otherwise.

  • A proposal to Endeavor Voyager customers

    If YPOW, like MPOW, is an Endeavor Voyager site, you’ve got some decisions ahead.  Francisco Partners, naturally, would like you to migrate to Aleph, and I have no doubt that Ex Libris is, as I write this, busily working on a means to make that easy for Voyager libraries to do.  But ILS migrations are painful, no matter how easy the backend process might be.  There’s staff training,  user training, managing new workflows, site integration; lots of things to deal with.  Also, your functionality may not be a 1:1 relationship to what you currently have.  How do you work around services you depended upon?

    Since soon our contracts with Endeavor Information Systems will be next to worthless, I propose, Voyager customers, that we take ownership of our systems.  For the price of a full Oracle (or SQL Server? — does Voyager support other RDBMSes?) license  (many of us already have this), we can get write permissions to our DB and make our own interfaces.  We wouldn’t need to worry about staff clients (for now), since we already have cataloging, circulation, acquisitions, etc. modules that work.  When we’re ready for different functionality, however, we can create a new middleware (in fact, I’m planning to break ground on this in the next two weeks) to allow for web clients or, even better, piggyback on Evergreen’s staff clients and let somebody else do the hard work.  If we had native clients in the new middleware, a library could use any database backend they wanted (just migrate the data from Oracle into something else).  The key is write access to the database.

    By taking ownership of our ILS, we can push developments we want, such as NCIP, a ‘Next Gen OPAC’, better link resolver integration, better metasearch integration, etc. without the pain of starting all over again (with potentially the same results, who is to say that whatever you choose as an ILS wouldn’t eventally get bought and killed off, as well?).  Putting my money (or lack thereof) where my mouth is, I plan on migrating Fancy Pants to use such a backend (read only db access, for now, we still have a support contract, after all).  I’m calling this project ‘Bon Voyage’.  After reading Birkin’s post on CODE4LIB, I would like to make a similar service for Voyager that would basically take the place of the Z39.50 server and access to the database.  Fancy Pants wouldn’t be integrated into Bon Voyage, it would just be another client (since it was always only meant as a stopgap, anyway).

    What we’ll have is a framework for getting at the database backend (it’d be safe to say this will be a rails project) with APIs to access bib, item, patron, etc. information.  Once the models are created, it will be relatively simple to transition to ‘write’ access when that becomes necessary.  Making a replacement for WebVoyage would be fairly trivial once the architecture is in place.  Web based staff clients would also be fairly simple.  I think EG staff client integration wouldn’t be too hard since it would just be an issue of outputting our data to something the EG clients want (JSON, I believe) and translating the client’s reponse.  That would need to be investigated more, however (I’m on paternity leave and not doing things like that right now 🙂

    Would anybody find this useful? 
    It seems the money we spend on an ILS could be better spent elsewhere.  I don’t think this would be a product we could distribute outside of the the current Voyager customer base (at least, not until it was completely native… maybe not even then- we’d have to work this out with Francisco Partners, I guess), but I think that that is big enough to be sustainable on its own.

  • Bringing up baby

    Well, we’re home. Things are going well. Guatemala was a lot of waiting… waiting for Che, waiting for the U.S. Embassy, waiting for his visa, waiting around the hotel, waiting for the airport, waiting in the airport, waiting to land, waiting at customs and immigration…

    He is wonderful, though, and we’re really enjoying things right now.

    I’m on leave until next Tuesday (after which I’ll be working from home until June), but since Che doesn’t talk much, it gives me lots of time to think while I’m rocking him to sleep or (even more) laying in bed awake wondering when he’s going to wake up next. With that, I’ve had some ‘work related ideas’ 🙂

    Photographic evidence of Che’s homecoming.

  • Che4Lib

    While the Lucene Preconference is starting, Selena and I will be working our way towards the airport to catch our flight to Guatemala City.

    As Karen Schneider delivers her opening keynote, we’ll be at the U.S. Embassy in Guatemala getting Isaac Mario Singer-Seymour’s passport signed.

    During the slot that I was supposed to speak about the Ümlaut and afterwards when the Metalib pilot group is picking Roy Tennant’s brain, Selena and I will be on our way back home, with our son.

    There couldn’t be a better reason to miss Code4Lib.  Have fun.

  • Pay no attention to the man behind the curtain

    port install rubygems

  • YYZ

    At the beginning of the month, I gave two presentations at the Ontario Library Association’s SuperConference. I had a good time. My first time to Toronto and it snowed. Although, with all due respect, Mr. Lee, YYZ totally sucked.
    My presentations were:

    Librarian’s Lib: Taking control of what’s ours

    The Communicat: contribute to the collective collection, comrade

    Some pictures I took while there

  • To build or to modify

    I readily admit that I have a bit of a NIH problem. A lot of this is laziness (hey, it’s a lot of work to figure out how somebody’s code works) coupled with the fact that I write fast married to the dilemma of if I modify this program to meet our needs, what happens when we need to upgrade?

    One place where I’ve really struggled with this is surrounding intranets. At Emory I built myStaffWeb specifically to handle situations like:

    1. Employee needs to submit timesheet.
    2. Timesheet needs to go to employee’s supervisor.
    3. If either of these tasks are late, emails need to be sent to the guilty party.
    4. The supervisor either approves the timesheet (sending it to HR) or rejects it (sending it back to the employee).

    Workflows like this are pretty commonplace for intranet-y tasks. The problem is that most off the shelf portal systems can’t handle them. Drupal, Joomla/Mambo, the Nukes all have horizontally based roles (members, editors, admins, etc.), which work fine for ‘content’ based sites, but lacks the granularity needed for ‘doing business’.

    I built something similar to myStaffWeb when I got to Tech to handle other, similar tasks (set GALILEO password, upload EAD finding aid, manage subject guides, etc.). For ‘custom’ type modules, such a system works well; I am able to build a new module quickly and based on user, group and role models set up workflows pretty easily.

    The problem is that as staff want to use more commonplace web tools (wikis, blogs, simple web pages, file management), it gets more and more complicated to keep up. The ability to draw from a large development community to help make your knowledgebase (say) or FAQ makes life a lot easier (and allows you to spend time working on things that help your public rather than your staff).

    There are, in fact, some portal/community CMSes that have both horizontal and vertical permissions (‘manager’ of this group), such as Plone and TikiWiki, so I had held out hope they would work for us.

    TikiWiki eliminated itself pretty quickly, however. It’s got an interface that only an engineer could love, is painfully slow, and uses the Galaxia workflow engine. I’m sure Galaxia is powerful and all, but it’s horribly over engineered for the simple workflows I need. Besides, most of our workflows are repetitive: employee -> supervisor -> task manager, to have to build that workflow in Galaxia every time for every app seems overkill.

    Plone seemed perfect in every way. Our sysadmin installed it for the library collaboration committee to experiment with and it appeared to solve our problems. Groups and roles within groups, blogs, wikis, content. It looked like it could handle our intranet and our public website. Assuming we committed to it.

    With every intranet I’ve developed, I have always looked at Zope. It seems to hold so much promise in this arena, but somehow, every time, something horrible goes awry. Zope (and therefore Plone) is an unholy beast and one of its biggest problems is that when something goes wrong, because it’s so alien (it uses its own webserver, its own database engine, it uses python in its own special way) there’s no body of knowledge to rely on to get you out of trouble. Nothing you’ve learned by using apache for the last 9 years helps you debug a Zope server problem. There may be ways to access the ZODB directly, but it’s not like being able to open phpMyAdmin and fixing a field value. Everytime I’ve dabbled with Zope, something bad has happened and I haven’t been able to fix it. And that sticks with you.

    We had one of those problems early on with Plone (the sysadmin ‘accidentally locked the keys in the car’, as I put it — a bad incompatibility with the Zope control panel and Plone’s CAS plugin), but we got around it, documented it, made a policy for plugins and forged ahead. Despite my reservations about Zope, I suggested we commit to using Plone for the intranet and the public web. I didn’t see any point in trying to maintain two CMS systems if Plone was going to work for the intranet.

    And then we found Plone’s fatal flaw. While working on a wiki to document how to troubleshoot the Umlaut while I was in Toronto, something happened and somehow I saved a much earlier revision of my page (losing about half my work). Since I had been saving my edits, I assumed I could just rollback to a previous save, but you know what happens when you assume. To my (and the sysadmin’s) horror, we realized that Plone has no concept of versioning.

    There are Plone products that deal with this, but none have ‘stable’ releases and, besides, depending on a product for such core functionality seems risky. When and if versioning is integrated into Plone core, will your product be compatible? Will you get stuck behind if a new version of Plone comes out and your particular product doesn’t work with it? This gaping hole in functionality (coupled with learning curve inherent in Plone to begin with) basically brought our Plone experiment to a grinding halt.

    So last week I started redesigning the existing intranet. I’ve migrated it to rails, and yet again I’m amazed at how quickly I can get core functionality running. Ruby/Rails is so much better suited to object model in our intranet than PHP was it makes this tremendously easier. Rather than worry about building wikis and blogs and whatnot, we’re just using off the shelf products that are remotely editable. For wikis, we’re using OddMuse, for blogs we’re using WP-MU. This way, I only have to make clients in the intranet to access these remotely.  In essence, the intranet aggregates services and manages permissions (this wiki is only available to the group or the user or to the library, etc.) and handles our specific needs, like timesheets and student employment applications and the like.

    So basically we’re finding a compromise between invented here and there.  Just in time for me to drop this project like a hot rock for a new position 🙂 (more on that later).

  • Excuses, Excuses

    Given our exciting news about Che, I will not be attending Code4Lib 2007.  It’s unclear whether or not we’ll be in Guatemala during the actual conference or just returned (I suppose there’s an outside chance that we won’t have gone, yet, but let’s not think about that), but I might as well give up my slot since we have a waiting list and my mind certainly wouldn’t be on it at any rate.

    Still, it was a little sad to see how quickly I was replaced.  That’s right, Singer, your 15 minutes are looking pretty 15 minutes ago.

  • Ãœps

    Argh.  I’ve recently migrated to using Opera on our desktop at home (long story, but basically the experiment with Ubuntu didn’t go over well; we went back to Windows; and IE7 is too slow to be considered useful in any capacity.  And Selena uses Firefox.).  While in the middle of a long post about how I’ve broken the Ümlaut (not the production or subversion versions — but the development server is FUBARed and a huge flaw has been exposed in its design), I managed to, with my stumpy, inaccurate man-fingers, hit some key combination by accident that caused me to leave my
    editing page and sent me to some ’Opera search page’.  Thanks!

    …pause while Ross saves…

    Anyway, a huge refactoring will be taking place (although, honestly, I don’t think it will take me very long).  The long and short of it is:  I was trying to make the Ümlaut database independent.  SQLite had a hard time with the Ümlaut’s liberal use of Marshaling objects in the database.  PostgreSQL wasn’t working with Rails and when I upgraded Rails…  all hell broke loose.  Basically my Marshal plan wasn’t going to work with the direction of Rails development.

    But, this has actually led me to a better way (and, I think, much more efficient) way to handle requests.

    So, stay tuned for Ü2.