Grails: Dynamically building criteria queries

** Update: Detached Criteria are probably the better way of achieving this type of DRY principle now.  They became available in Grails 2.0; I wrote this post while using Grails 1.x. **

** Update 2: DetachedCriteria have some limitations. I wrote a cleaned up and expanded version of what this post discusses: Criteria Aggregator: Dynamic Criteria Queries in Grails **

Ok, I’ve been exploring using Groovy and Grails at work to make our team more efficient. Java is great and all, but it’s a terrible language for web development. For some quick info: A group of people decided Java could be a good web development language if it weren’t so Java-y. So they wrote Groovy. It’s fully compatible with Java, but is actually a dynamically-typed scripting language built on top of Java. Great, now we have a good web language with all the support and power of Java. Groovy compiles to the same bytecode as Java, because Java is Groovy is Java.

Well, then Ruby took the web development world by storm with the Rails architecture. Finally web developers could focus on the actual interesting part of the job rather than continually writing frameworks to run their sites. It was such a good idea that it was very quickly mirrored into other languages. Python got Django (my personal favorite and the future of the 100 Hour Board) and Groovy got Grails.

Anyway, if you’ve found this post by Googling you probably know all that so let’s get to the meat of the matter.

In my effort to learn Groovy and Grails I wanted to be able to build up Criteria Queries dynamically. Django lets you do this incredibly easily by passing around queryset objects that have lazy loading. Grails interfaces with Hibernate using some convenient and easy closure work with access to the Hibernate Criteria Builder. So you can easily and quickly define your query but only within that closure that gets passed to the builder. I wanted to be able to piece my query together from other pieces in order to adhere to the Don’t Repeat Yourself (DRY) principle.

In my searching I came upon a blog post by Geoff Lane at zorched.net entitled “DRYing Grails Criteria Queries“. His method involves extending the metaclass of the HibernateCriteriaBuilder object using Groovy’s expando abilities. This is a decent solution, but not incredibly flexible. I needed something better.

For 3 days I relentlessly scoured the Internet searching for answers, I coded all sorts of tests, I digested the source code of the Grails HibernateCriteriaBuilder class, I poked, I prodded, I made wild accusations. All in vain.

I decided that one of the following conclusions must be true:
1. It was so obvious that I just couldn’t see it.
2. I was too stupid to come up with a good solution.
3. I was asking the wrong question.
4. I needed more experience with Groovy and working with closures.

Now, I’m not entirely convinced that #3 isn’t true, but #4 also happened to be true. I found my solution this morning when I actually paid attention to what the error message was really telling me rather than just “it didn’t work”.

So, without further ado, here is how you can achieve DRYness with Grails when dynamically building criteria queries, without dropping all the way back into Java.

For the sake of this example let’s assume we have an Account class with the properties “accountType”, “accountAction”, and “owner”.

Our automatically generated code to list objects would look something like this (in our AccountController.groovy file):

def list = {
  params.max = Math.min( params.max ? params.max.toInteger() : 20,  100)
  params.offset = params.offset ? params.offset.toInteger() : 0

  def results = Account.createCriteria().list(max:params.max, offset:params.offset) {
    order(params.sort ? params.sort : "id", params.order ? params.order : "asc")
  }
  [ accountInstanceList: results, accountInstanceTotal: results.totalCount ]
}

That’s all well and good, but let’s add the ability to sort based on the name of the account owner, and display only accounts belonging to a specific user:

def list = {
  params.max = Math.min( params.max ? params.max.toInteger() : 20,  100)
  params.offset = params.offset ? params.offset.toInteger() : 0
  def req_owner = User.get(params.id)

  def results = Account.createCriteria().list(max:params.max, offset:params.offset) {
    if (params.sort == 'userName') {
      owner {
        if (req_owner) {eq("id", req_owner.id)}
     order("lastName", params.order ? params.order : "asc")
     order("firstName", params.order ? params.order : "asc")
      }
    } else {
      if (req_owner) {owner {eq("id", req_owner.id)}}
      order(params.sort ? params.sort : "id", params.order ? params.order : "asc")
    }
  }
  [ accountInstanceList: results, accountInstanceTotal: results.totalCount ]
}

Still fine, but what if we want to build off of this for different controller methods? If I want a page that does the same thing, but limits the display to only accounts of a certain account type I can use the method described in the link above by Geoff Lane. But suppose I want to further provide another page that limits it further or several pages that provide just slightly different listings based on criteria changes. Perhaps there is a great easy way to do this in Grails, and if so, please comment and tell me, because I couldn’t find it.

Here’s how I do it. First let’s pull the code we have to a reusable location. We make a self-standing closure and then curry the “params” variable to it:

def account_list = {params ->
  def req_owner = User.get(params.id)
  if (params.sort == 'userName') {
    owner {
      if (req_owner) {eq("id", req_owner.id)}
      order("lastName", params.order ? params.order : "asc")
      order("firstName", params.order ? params.order : "asc")
    }
  } else {
    if (req_owner) {owner {eq("id", req_owner.id)}}
    order(params.sort ? params.sort : "id", params.order ? params.order : "asc")
  }
}

def list = {
  params.max = Math.min( params.max ? params.max.toInteger() : 20,  100)
  params.offset = params.offset ? params.offset.toInteger() : 0

  def results = Account.createCriteria().list(max:params.max, offset:params.offset, account_list.curry(params))
  [ accountInstanceList: results, accountInstanceTotal: results.totalCount ]
}

The trouble is when we try to combine self-standing closures.

Everytime I tried something that seemed like it should work I’d get errors about “AccountController.isNotNull not applicable for arguments…” or whatever it said. I didn’t think hard enough about what it was really saying to me. The additional closure pieces were getting the wrong scope for whatever reason and instead of calling methods on the CriteriaBuilder object they were calling them on the AccountController object.

Here’s the solution:

def crit_a = {crit ->
  crit.accountAction { 
    ne("name", "None")
  }
}

def crit_b = {crit ->
  crit.accountAction {
    ne("name", "Renew")
  }
} 

def account_list = {crit, params ->
  def req_owner = User.get(params.id)
  if (params.sort == 'userName') {
    crit.owner {
      if (req_owner) {eq("id", req_owner.id)}
      order("lastName", params.order ? params.order : "asc")
      order("firstName", params.order ? params.order : "asc")
    }
  } else {
    if (req_owner) {crit.owner {eq("id", req_owner.id)}}
    crit.order(params.sort ? params.sort : "id", params.order ? params.order : "asc")
  }
}

def list = {
  params.max = Math.min( params.max ? params.max.toInteger() : 20,  100)
  params.offset = params.offset ? params.offset.toInteger() : 0

  def critBuilder = Account.createCriteria()
  def crit_closure = {
    account_list.curry(critBuilder, params)()
    crit_a.curry(critBuilder)()
    crit_b.curry(critBuilder)()
  }

  def results = critBuilder.list(max:params.max, offset:params.offset, crit_closure)
  [ accountInstanceList: results, accountInstanceTotal: results.totalCount ]
}

You get a reference to the CriteriaBuilder object using Account.createCriteria(). Note that you get a CriteriaBuilder object not a Criteria object like much of the documentation suggests. You curry this to the closures you want to use, and call them inside a new closure which you pass to the CriteriaBuilder object’s “list” method. And it works! So now you can dynamically build up your criteria queries!

America’s Broken Laws

Do you remember Real Player? Huge in the late 1990s. They provided software for streaming audio/video over the Internet before Macromedia blew them out of the water with Flash. They’ve been quiet for a while, but recently they re-emerged with their latest product: RealDVD. It is software that allows you to make a backup of a DVD.

They knew that this was going to get them sued by the MPAA, and they were looking for the fight. As much as I (and millions of others) hated RealPlayer back in the 1990’s I (and millions of others) have been on their side in this battle.

So what is the battle? DVDs are “protected” with the Content Scramble System (CSS). In 1999 DVD Jon gained his name by cracking this system; allowing anyone to access the data stored on DVDs without paying the fee required to get the code to decrypt the content. Yes, that’s right. Any DVD player you’ve ever used; whether in your laptop, desktop, or connected to the tv; was only allowed to be produced after the manufacturer ponied up the cash for the license to legally decrypt the CSS and agreed to the demands that they wouldn’t produce a product which allows the consumer to make a copy of the DVD.

So when DVD Jon reverse-engineered CSS, and let the cat out of the bag, the MPAA was not happy. Luckily for the MPAA they had already managed to pass the Digital Millennium Copyright Act (DMCA). Section 1201 (2) Says:

`(2) No person shall manufacture, import, offer to the public, provide, or otherwise traffic in any technology, product, service, device, component, or part thereof, that–
`(A) is primarily designed or produced for the purpose of circumventing a technological measure that effectively controls access to a work protected under this title;
`(B) has only limited commercially significant purpose or use other than to circumvent a technological measure that effectively controls access to a work protected under this title; or
`(C) is marketed by that person or another acting in concert with that person with that person’s knowledge for use in circumventing a technological measure that effectively controls access to a work protected under this title.

In short: If there is any kind of copy-protection measure on a piece of media then it is illegal to access the content without paying the proper people for access (and since the people authorizing access won’t authorize any use allowing duplication we have our problem).

DVD Jon’s work is illegal under the DMCA. Fortunately for DVD Jon, he lives in Norway and the MPAA can’t really do much about it. The bigger issue for the MPAA is that the code necessary to break the encryption on a DVD is so simple it can be written on a t-shirt. Not exactly a big hurdle for people that want to use it.

So why is this an argument at all in the first place? Something is illegal, so you shouldn’t do it, right? Well, the problem is this little notion of Fair Use (also see the Electronic Frontier Foundation’s FAQ). The Fair Use doctrine says you can make a personal back-up copy of content you own.

Fair Use says you can make a backup copy of any content you own, the DMCA says it’s illegal to make that copy if the content has any type of copy-protection system in place. Take a guess which side wins in these arguments. I’ll give you a hint, it’s not us, the individual citizens of the country. If we want to make a back-up of The Fox and the Hound so that when the DVD gets all scratched and destroyed we don’t have to buy it again, we can’t.

This is the current state of the law in the United States. Absolutely ridiculous and inconsistent. The RealDVD case was decided today by U.S. District Judge Marilyn Hall Patel in San Francisco (ruled against Real). In her remarks she made this absurdity very clear:

So while it may well be fair use for an individual consumer to store a backup copy of a personally owned DVD on that individual’s computer, a federal law has nonetheless made it illegal to manufacture or traffic in a device or tool that permits a consumer to make such copies,

(For a complete article on the matter see Wired’s Judge Rules DVD-Copying Software Is Illegal)

Django – Cronjobs Made Easy!

For those that have no interest in reading about my nerd-ventures, you can stop reading this post now.

If you’re still reading, don’t say I didn’t warn you.

As has been mentioned previously (mainly on my previous blog), I’ve been doing a fair bit of side project work using the Django Framework. Sadly, the out-of-the-box Django doesn’t provide a solution for running cronjobs (for tasks that need to be run within the Django environment).

Since that’s a fairly common requirement I didn’t think it was going to be a big deal, but there wasn’t a really solid solution out there. There are a few different attempts, but they each have some limitation. There’s django-cron but that just skips over the native cron entirely, which I felt was a bit extreme. Cron can already do a good job of waking up and running a command, so duplicating that functionality doesn’t seem necessary. It also self-declares that it is designed for frequent tasks (hourly or more frequently), which doesn’t work for me. Tasks on the Board need to be able to run from a minute scale to a daily scale and beyond.

Then I found this guy’s method, which works, but I’d like a little more integration. That way when developing apps the “go add cron job” isn’t a separate step. I want the job information to right in with the rest of my app information. That way I can see what should be happening and when.

That’s when I came across Django-Chronograph. This solution was 95% of what I wanted. It provides a nice interface to the system to monitor your jobs and view logs. It requires only a single crontab entry. It uses the iCalendar style of task declaration so you have total control of when your jobs run. However, it is limited to running commands through the Django Management system. I wanted something a little more programmatic. Such that I could just point at whatever function I wanted for my jobs.

So I took Django-Chronograph and started my modifications. The result is Django Cron Manager. Setup is very similar to using the Admin system. You call the cron_manager.autodiscover() function from your urls.py file. This goes out and inspect your installed apps and registers any Cron Jobs they declare. Then, using the guts of Django-Chronograph, it keeps track of these jobs in the database and monitors when they need to run.

I’m planning on posting all the code with an example at some point, but I’m going to try to get in touch with Weston (the guy who wrote Django-Chronograph) to see if he just wants to roll my changes into his system permanently. If you stumble upon this post and the changes aren’t in Django-Chronograph, and I haven’t provided any further information. Just leave a comment that you’re interested in the code with a way to contact you and I’ll get something to you.

*** Update ***
I’ve posted the code here: http://code.google.com/p/django-chronograph/issues/detail?id=15

Ubuntu 9.04 Touchpad Scroll Region

Jess has switched her laptop to Ubuntu and in the process there are some tweaks to be done. One of them was that her touchpad has a little area set apart for scrolling and the default option on Ubuntu was allowing scrolling for a small amount past that set apart area. Unfortunately, they’ve just updated those settings to be part of the HAL device settings rather than the Xorg.conf settings. So now there is not a nice clean GUI configuration program.

After some messing around I was able to figure it out though. You need to create an FDI policy in the directory “/etc/hal/fdi/policy”. It doesn’t seem to matter what you call this as long as it ends in “.fdi”. So I called mine “touchpad.fdi”. In this file we put:

<?xml version="1.0" encoding="ISO-8859-1"?>
<deviceinfo version="0.2">
<device>
<match key="input.x11_driver" string="synaptics">
 <merge key="input.x11_options.RightEdge" type="string">5900</merge>
</match>
</device>
</deviceinfo>

(For more information on how to form this file see: https://wiki.ubuntu.com/X/Config/Input.)

If you follow these instructions and things don’t seem to work, you might try switching the encoding option from “ISO-8859-1” to “UTF-8”.

Anyway, save that text into a file like “/etc/hal/fdi/policy/touchpad.fdi”. The next step (which was left out of 99% of the information I found online and made me very frustrated) is to restart HAL:

sudo /etc/init.d/hal restart

After HAL restarts, log out and then log back in.

If it found and read your file properly something should have changed. If you want to test to see if things are working at all you can try setting the RightEdge value (5900 in the above) to something small (I’ll warn you that this will probably turn your entire touchpad into a vertical scroll area, however you can do some sneakiness to recover, first you can still access a terminal by accessing your menus using ALT+F1, and you can access the shutdown menu (not the logout menu sadly) with ALT+CTRL+DEL).

You’ll need to play with that value in order to get it to work for you. I tried to be clever and calculate it from some information, but it didn’t work, so [shrug] just play with it. I changed things about 200 at a time which was enough of a change to notice.

I hope this helps someone out.