{"id":396,"date":"2010-08-03T16:03:26","date_gmt":"2010-08-03T21:03:26","guid":{"rendered":"http:\/\/dilettantes.code4lib.org\/blog\/?p=396"},"modified":"2010-08-03T16:03:26","modified_gmt":"2010-08-03T21:03:26","slug":"faceted-search-on-a-shoestring","status":"publish","type":"post","link":"https:\/\/rossfsinger.me\/blog\/2010\/08\/faceted-search-on-a-shoestring\/","title":{"rendered":"Faceted Search on a Shoestring"},"content":{"rendered":"

There are any number of reasons that you can attribute to Solr<\/a>‘s status as the standard bearer of faceted full-text searching: \u00c2\u00a0it’s free, fast, works shockingly well out of the box without any tweaking, has a simple and intuitive HTTP\u00c2\u00a0API (making it available in the programming language of your choice) and is, by far, the easiest “enterprise-level” application to get up and running. \u00c2\u00a0None of its “competitors” (Sphinx<\/a>, Xapian<\/a>, Endeca<\/a>, etc.), despite any individual advantages they might have, can claim all of these features, which goes a long way towards explaining Solr’s popularity.<\/p>\n

The library world has definitely taken a shine to Solr: \u00c2\u00a0from discovery interfaces like VuFind<\/a> and Primo<\/a>, to repositories like Fedora<\/a>, to full-text aggregators like Summon<\/a>, you can find Solr under the hood of most of the hot products and services available right now. \u00c2\u00a0The fact that a library can install VuFind and have a slick, jaw-droppingly powerful OPAC-replacement that puts their legacy interface to shame in about an hour<\/em> is almost completely the by-product of Solr’s amazing simplicity to get up and running. \u00c2\u00a0It’s no wonder why so many libraries are adopting it (compare it to SOPAC<\/a>, also built in PHP and about as old, but uses Sphinx for the full-text indexing and is hardly ever seen in the wild).<\/p>\n

Without a doubt, Solr is pretty much a no-brainer if you are able to run Jetty (or Tomcat or JBoss or Glassfish or whatever): \u00c2\u00a0with enough hardware, Solr can scale up to pretty much whatever your need might be. \u00c2\u00a0The problem (at least the problem in my mind) is that Solr doesn’t scale down<\/em> terribly well. \u00c2\u00a0If you host your content from a cheap, shared web hosting provider or a VPS, for example, Solr is not available or not practical (it doesn’t live in small memory environments well). \u00c2\u00a0The hosted Solr<\/a> options<\/a> are fairly expensive and while there are cheap, shared web hosting providers that do provide Java Application Servers<\/a>, switching vendors to provide faceted search for your mid-size Drupal<\/a> or Omeka<\/a> site might not be entirely practical or desirable.<\/p>\n

I find myself proof-of-concept-ing a lot of hacks to projects like VuFind, Blacklight<\/a>, Kochief<\/a> and whatnot and run these things off of my shared web server. \u00c2\u00a0It’s older, underpowered and only has 1GB of RAM. \u00c2\u00a0Since I’m not running any of these projects in production (just really making things available for others to see), it was really annoying to have Solr gobbling up 20% of the available RAM for these little pet projects. \u00c2\u00a0What I wanted was something that acted more or less like Solr when you pointed an application that expected Solr to be there, but I wanted it to have a small footprint that could run (almost) anywhere and more or less disappear when it was idle.<\/p>\n

So it was for this scenario that I wrote CheapSkate<\/a>: a Solr emulator written in Ruby. \u00c2\u00a0It uses Ferret<\/a>, the Ruby port of Lucene, as the full-text indexing engine and Sinatra<\/a> to supply the HTTP API. \u00c2\u00a0Ferret is fast, scales quite well and responds to the same search syntax as Solr, so I knew it could handle the search aspect pretty easily. \u00c2\u00a0Faceting (as can be expected) proved the harder part. \u00c2\u00a0Originally, I was storing the values of fields in an RDBMS and using that to provide the facets. \u00c2\u00a0Read performance was ok, although anything over 5,000 results would start to bog down – the real problem was the write performance, which was simply woeful. \u00c2\u00a0Part of the issue was that this design was completely schemaless: \u00c2\u00a0you could send anything to CheapSkate and facet on any field, regardless of size. \u00c2\u00a0It also tried to maintain the type of the incoming field value: \u00c2\u00a0dates were stored as dates, numbers stored as integers and so on. \u00c2\u00a0Basically the lack of constraints made it wildly inefficient.<\/p>\n

Eventually, I dropped the RDBMS component, and started playing around Ferret’s terms capabilities<\/a>. \u00c2\u00a0If you set a particular field to be untokenized, your field values appear exactly as you put them in. \u00c2\u00a0This is perfect for faceting (since you don’t want stemming and whatnot on your query filters and your strings aren’t normalized or downcased or anything so they look right in the UI) and is basically the same thing Solr itself does. \u00c2\u00a0Instead of a schema.xml, CheapSkate has a schema.yml, but it works essentially the same way: \u00c2\u00a0you define your fields, what should be tokenized (that is, which fields allow full-text search) or not (i.e. facet fields) and what datatype the field should be.<\/p>\n

CheapSkate doesn’t support all of the field types that Solr does, but it supports strings, numbers, dates and booleans.<\/p>\n

One neat thing about Ferret is that you can pass a Ruby Proc to the search method<\/a> as a search option. \u00c2\u00a0This proc then has access to the search results as Ferret is finding them. \u00c2\u00a0CheapSkate uses this find the terms in the untokenized fields for each search hit, throws them in a Hash and generates a hit count for each term. \u00c2\u00a0This is a lot<\/em> faster than getting all the document ids from the search, looping them and generating your term hash after<\/em> the search is completed. \u00c2\u00a0That said, this is still definitely the bottleneck for CheapSkate. \u00c2\u00a0If the search result has more than 10-15,000 hits, performance begins to get pretty heavily impacted by grabbing the facets. \u00c2\u00a0I’m not terribly<\/em> concerned by this, data sets with search results in the 20,000+ range start to creep into the “you would be better off just using Solr” domain. \u00c2\u00a0For my proofs-of-concepts, this has only really raised its head in VuFind when filtering on something like “Book” (with no search terms) for a 50,000 record collection. \u00c2\u00a0What I mean to say is, this happens for fairly non-useful searches.<\/p>\n

Overall, I’ve been pretty happy with how CheapSkate is working. \u00c2\u00a0For regular searching it does pretty well (although, like I said, I’m not trying to run a production discovery system that pleases both librarians and users). \u00c2\u00a0There’s a very poorly designed “more like this” handler that really needs an overhaul and there is no “did you mean” (spellcheck). \u00c2\u00a0This hasn’t been a huge priority, because I don’t really like the spellcheck in Solr all that much, anyway. \u00c2\u00a0That said, if somebody really wanted this and had an idea of how it would be implemented in Ferret, I’d be happy to add it.<\/p>\n

Ideally, I’d like to see something like CheapSkate in PHP using Zend_Search_Lucene<\/a>, since that would be accessible to virtually everybody, but that’s a project for somebody else.<\/p>\n

In the meantime, if you want to see some examples of CheapSkate in action:<\/p>\n