{"id":289,"date":"2009-03-19T21:33:43","date_gmt":"2009-03-20T02:33:43","guid":{"rendered":"http:\/\/dilettantes.code4lib.org\/?p=289"},"modified":"2009-03-19T21:33:43","modified_gmt":"2009-03-20T02:33:43","slug":"in-search-of-a-bag-of-holding","status":"publish","type":"post","link":"https:\/\/rossfsinger.me\/blog\/2009\/03\/in-search-of-a-bag-of-holding\/","title":{"rendered":"In search of a bag of holding"},"content":{"rendered":"<p>Let me start this by saying this is not a criticism or a rant against any of the technologies I am about to mention.\u00c2\u00a0 The problems I am having are pretty specific and the fact that I am intentionally trying to use &#8220;off the shelf&#8221; commodity projects to accomplish my goal complicates things.\u00c2\u00a0 I realize when <a href=\"http:\/\/twitter.com\/rsinger\/status\/1348974979\" target=\"_blank\">I tweet things like this<\/a>, it&#8217;s not helpful (because there is zero context).<\/p>\n<p>I&#8217;ve been in a bit of a rut this week.\u00c2\u00a0 Things were going ok on Monday, when I got the Jangle connector for <a href=\"http:\/\/www.reservesdirect.org\/\" target=\"_blank\">Reserves Direct<\/a> working, announced and started <a href=\"http:\/\/groups.google.com\/group\/jangle-discuss\/browse_thread\/thread\/255f41088bba9434\" target=\"_blank\">generating some conversation around how to model course reserves<\/a> (and their ilk) in Jangle.\u00c2\u00a0 However, this left me without anything specific to work on.\u00c2\u00a0 I have a general, hand-wavy, project that I am supposed to be working on to provide a simple, general digital asset management framework that can either work on the <a href=\"http:\/\/n2.talis.com\/\" target=\"_blank\">Platform<\/a> or with a local repository like <a href=\"http:\/\/fedoracommons.org\/\" target=\"_blank\">Fedora<\/a>, depending on the institution&#8217;s local needs or policies.\u00c2\u00a0 More on this in some other post.\u00c2\u00a0 The short of it is, I need to gather some general requirements before I can begin something like this in earnest, which led me to revive an old project.<\/p>\n<p>When Jangle first started, about 15 months ago, <a href=\"http:\/\/townx.org\/\" target=\"_blank\">Elliot<\/a> and I felt we needed what we called &#8220;<a href=\"http:\/\/code.google.com\/p\/jangle\/wiki\/BaseConnector\" target=\"_blank\">the Base Connector<\/a>&#8220;.\u00c2\u00a0 The idea here was that there were always going to be situations where a developer doesn&#8217;t have direct, live access to a system&#8217;s data, and they would need a surrogate external database to work with.\u00c2\u00a0 The Base Connector was an attempt to provide an out of the box application that could simulate the basics of an ILS and be populated with the sorts of data you would get from commandline &#8216;report&#8217; type applications.\u00c2\u00a0 The sort of thing you can cron and write out to a file on your ILS server.\u00c2\u00a0 Updates in catalog records.\u00c2\u00a0 Changes in user status.\u00c2\u00a0 Transactions.\u00c2\u00a0 That sort of thing.<\/p>\n<p>After the amount of interest at <a href=\"http:\/\/code4lib.org\/conference\/2009\/\" target=\"_blank\">Code4lib<\/a> in Janglifying <a href=\"http:\/\/www.iii.com\/products\/millennium_ils.shtml\" target=\"_blank\">III Millenium<\/a>, I decided to revisit the concept of the Base Connector.\u00c2\u00a0 Millenium&#8217;s (and to an extent, Unicorn&#8217;s, and there are no doubt others) lack of consistent database access, makes it a good candidate for this duplicate database.\u00c2\u00a0 I was hoping to take a somewhat different approach to this problem than Elliot and I had originally tried, however.\u00c2\u00a0 I was hoping to be able to come up with something:<\/p>\n<ol>\n<li>More generically &#8220;Jangle&#8221; and less domain specific to ILSes<\/li>\n<li>Easy to install<\/li>\n<li>Customizable with a simple interface<\/li>\n<li>Something, preferably, that could be taken mostly &#8220;off the shelf&#8221;, where the only real &#8220;coding&#8221; I had to do was to get the library data in and the connector API out.\u00c2\u00a0 I was hoping all &#8220;data model&#8221; and &#8220;management&#8221; stuff could be taken care of already.<\/li>\n<\/ol>\n<p>In my mind, I was picturing using a regular CMS for this, although it needed to be able to support customized fields for resources.<\/p>\n<p>Here is the rough scenario I am picturing.\u00c2\u00a0 Let&#8217;s say you have an ILS that you don&#8217;t have a lot of access to.\u00c2\u00a0 For your external &#8216;repository&#8217;, you&#8217;ll need it to be able to accomodate a few things.<\/p>\n<ul>\n<li>Resources will need not just a title, an identifier and the MARC record, but also have fields for ISBN, ISSN, OCLC number, etc.\u00c2\u00a0 They&#8217;ll also need some sort of relationship to the Items and Collections they&#8217;re associated with.<\/li>\n<li>Actors could be simple system user accounts, but they&#8217;ll need first names and last names and whatnot.<\/li>\n<li>Collections, I assume, can probably be contrived via tags and whatnot.<\/li>\n<li>The data loading would probably need to be able to be done remotely via some commandline line scripting.<\/li>\n<\/ul>\n<p>I decided to try three different CMSes to try to accomplish this:\u00c2\u00a0 <a href=\"http:\/\/drupal.org\/\" target=\"_blank\">Drupal<\/a>, <a href=\"http:\/\/plone.org\/\" target=\"_blank\">Plone<\/a> and <a href=\"http:\/\/www.daisycms.org\/\" target=\"_blank\">Daisy<\/a>.\u00c2\u00a0 I&#8217;ll go through each and where I ran into a snag for each.\u00c2\u00a0 I want to reiterate here that I know next to nothing about any of these.\u00c2\u00a0 My problems are probably not shortcomings of the project themselves, but more due to my own ignorance.\u00c2\u00a0 If you see possible solutions to my issues (or know of other possible projects that fit my need even better) please let me know.\u00c2\u00a0 This is a cry for help, not a review.<\/p>\n<p><strong>Drupal<\/strong><\/p>\n<p>One of the reasons I targeted Drupal is that it&#8217;s easy to get running, can run on cheap shared hosting, <a href=\"http:\/\/listserv.uic.edu\/archives\/drupal4lib.html\" target=\"_blank\">has quite a bit of traction in libraries<\/a> and has <a href=\"http:\/\/drupal.org\/node\/101723\" target=\"_blank\">CCK<\/a>.\u00c2\u00a0 I actually got the farthest with Drupal in this capacity.\u00c2\u00a0 With CCK, I was able to, in the native Drupal interface, build content types for Resources and Items.\u00c2\u00a0 For Actors, I had just planned on using regular user accounts (since then I could probably piggyback off of the contributed authentication modules and whatnot).\u00c2\u00a0 Collections would be derived from Taxonomies.<\/p>\n<p><em>Where things went wrong:<\/em><\/p>\n<p>My desire is to decouple the &#8216;data load&#8217; aspect of the process from the &#8216;bag of holding&#8217; itself.\u00c2\u00a0 What I&#8217;m saying is that I would prefer that the MARC\/borrower\/item status\/etc. load not be required to be built in Drupal module, but, instead, be able to be written in whatever language the developer is comfortable with and a simple way of getting that data into the bag of holding.<\/p>\n<p>There are only two ways that I can see to use an external program to get data into Drupal:<\/p>\n<ol>\n<li>Use the XMLRPC interface<\/li>\n<li>Simulate the content creation forms with a scripted HTTP client.<\/li>\n<\/ol>\n<p>I&#8217;m not above number two, but I would prefer not to if there&#8217;s a better way available.\u00c2\u00a0 The problem is that I can find almost zero documentation on the XMLRPC service.\u00c2\u00a0 What &#8216;calls&#8217; are available?\u00c2\u00a0 How do I create a Resource content type?\u00c2\u00a0 How do I relate that to a user or an Item?\u00c2\u00a0 I have no idea where to look.\u00c2\u00a0 I don&#8217;t actually even know if the fields I created will be searchable (which was the whole point of making them).<\/p>\n<p>Drupal seems promising for this, but I don&#8217;t know where to go from here.<\/p>\n<p><strong>Plone<\/strong><\/p>\n<p>I really thought Plone was going to be a winner.\u00c2\u00a0 It&#8217;s completely self-contained (framework, webserver and database all rolled into one installer) and based on an OODB.\u00c2\u00a0 Being Python based, I feel I can fall back on Python to build the scripts to actually do the dirty work of massaging and loading the data.\u00c2\u00a0 The downside to Plone (and I have looked eye-to-eye with this downside before) is that it and <a href=\"http:\/\/zope.org\/\" target=\"_blank\">Zope<\/a> are total voodoo.<\/p>\n<p>It didn&#8217;t take me long to run into a brick wall with Plone.\u00c2\u00a0 I installed version 3.2.1 thanks to the <a href=\"http:\/\/plone.org\/products\/plone\" target=\"_blank\">handy OSX installer<\/a> and got it up and running.<\/p>\n<p>And then I couldn&#8217;t figure out what to do next.\u00c2\u00a0 I <em>think<\/em> I want <a href=\"http:\/\/plone.org\/products\/archetypes\" target=\"_blank\">Archetypes<\/a>.\u00c2\u00a0 I followed the <a href=\"http:\/\/plone.org\/products\/archetypes\/documentation\/link\/old-archetypes-docs\" target=\"_blank\">(outdated)\u00c2\u00a0 instructions<\/a> to install it.\u00c2\u00a0 I see Archetypes stuff in the Zope control panel.\u00c2\u00a0 However, I never see anything in Plone.\u00c2\u00a0 I Google.\u00c2\u00a0 Nothing.\u00c2\u00a0 Feeling that it must be there and I&#8217;m just missing something <a href=\"http:\/\/www.devshed.com\/c\/a\/Zope\/Plone-Content-Types-With-Archetypes\/\" target=\"_blank\">I follow this tutorial<\/a> to start building new content types.\u00c2\u00a0 I build a new content type.\u00c2\u00a0 <em>It<\/em> doesn&#8217;t show up in the Plone quick installer.\u00c2\u00a0 Nothing in the logs.\u00c2\u00a0 I Google.\u00c2\u00a0 Nothing.<\/p>\n<p>Nothing frustrates me more than software making me feel like total dumbass.<\/p>\n<p>I am at the point where I think Plone might be up to the task, but I don&#8217;t have the interest, time or energy to make it work.\u00c2\u00a0 At the end of the day, I&#8217;m still not entirely sure that it would meet my basic criteria of the &#8216;content type&#8217; being editable within the native web framework anyway.\u00c2\u00a0 I also have no idea if my plan of loading the data via an external Python (or, even better, Ruby) script is remotely feasible.<\/p>\n<p>Plone got the brunt of my disgruntled tweeting.\u00c2\u00a0 This is mainly due my frustration at seeing how well Plone would fit it my vision and being able to get absolutely nowhere towards realizing that goal.<\/p>\n<p><strong>Daisy<\/strong><\/p>\n<p>What, you&#8217;ve never heard of it?\u00c2\u00a0 I have a <a href=\"http:\/\/journal.code4lib.org\/articles\/24\" target=\"_blank\">history with Daisy<\/a>, and I know, without a doubt, it could serve my needs.\u00c2\u00a0 The problem with Daisy is that it has <em>a lot<\/em> of working parts.\u00c2\u00a0 To do what I want, you need both the data repository <em>and<\/em> the wiki running, as well as MySQL.\u00c2\u00a0 On top of that, some external web app would need to actually do the Jangle stuff (and, this would most likely be Ruby\/Sinatra) interacting with the HTTP API.\u00c2\u00a0 This is a lot of running daemons.\u00c2\u00a0 A lot of daemons that might not be running at any given time which would break everything.\u00c2\u00a0 Daisy is a lot of things, but it&#8217;s not &#8216;self-contained&#8217;.<\/p>\n<p>This is not a criticism.\u00c2\u00a0 If I was running a CMS, this would be ok.\u00c2\u00a0 When I was developing the Communicat, this was ok.\u00c2\u00a0 Those are <em>commitments<\/em>.\u00c2\u00a0 Projects that you think, &#8220;ok, I&#8217;m going to need to invest some thought and energy into this&#8221;.<\/p>\n<p>The bag of holding is a stop-gap.\u00c2\u00a0 &#8220;I need to use this until I can figure out a <em>real<\/em> way to accomplish what I need&#8221;.\u00c2\u00a0 Maybe it&#8217;s the ultimate production service.\u00c2\u00a0 That&#8217;s fine, but it needs to scale down as far as it scales up.\u00c2\u00a0 I literally want something that somebody can easily get running anywhere, quickly and start Jangling.<\/p>\n<p>If anybody has any recommendations on how I can easily get up and running with any of the above projects, please let me know.<\/p>\n<p>Alternately, if anybody knows something else, a simple, remotely accessible dynamic, searchable data store, <em>definitely enlighten me<\/em>!\u00c2\u00a0 I realize the irony of this plea, given who I work for, but the idea here is for something not cloud based, since I would like for the user to be able to load in their sensitive patron data without having to submit it to some third party service.\u00c2\u00a0 There&#8217;s also the fact that there&#8217;s no front end that I can just &#8216;plug in&#8217; to manage the data.<\/p>\n<p>If I can&#8217;t get anything off the shelf working, I think I&#8217;ll be reduced to writing something simple in Merb or Sinatra with CouchDB or Solr or something.\u00c2\u00a0 I was really hoping to have to avoid doing this, though.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Let me start this by saying this is not a criticism or a rant against any of the technologies I am about to mention.\u00c2\u00a0 The problems I am having are pretty specific and the fact that I am intentionally trying to use &#8220;off the shelf&#8221; commodity projects to accomplish my goal complicates things.\u00c2\u00a0 I realize [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[58,57,47,56],"tags":[],"class_list":["post-289","post","type-post","status-publish","format-standard","hentry","category-daisycms","category-drupal","category-jangle","category-plone"],"_links":{"self":[{"href":"https:\/\/rossfsinger.me\/blog\/wp-json\/wp\/v2\/posts\/289","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rossfsinger.me\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rossfsinger.me\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rossfsinger.me\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rossfsinger.me\/blog\/wp-json\/wp\/v2\/comments?post=289"}],"version-history":[{"count":5,"href":"https:\/\/rossfsinger.me\/blog\/wp-json\/wp\/v2\/posts\/289\/revisions"}],"predecessor-version":[{"id":294,"href":"https:\/\/rossfsinger.me\/blog\/wp-json\/wp\/v2\/posts\/289\/revisions\/294"}],"wp:attachment":[{"href":"https:\/\/rossfsinger.me\/blog\/wp-json\/wp\/v2\/media?parent=289"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rossfsinger.me\/blog\/wp-json\/wp\/v2\/categories?post=289"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rossfsinger.me\/blog\/wp-json\/wp\/v2\/tags?post=289"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}