Cluster Support for Genetic Programming

While John has been working on Erlush (more of a rabbit hole than he expected, I’m sure), I’ve been bouncing around between a few different ideas for interacting with Erlush from other language environments and for launching Erlush programs across cluster nodes. Coming up to the end of the semester, here’s a look at the current state of affairs in both of these areas:

• Foreign language interfaces: I futzed around for a while with the (probably doomed-from-the-get-go) idea of using standard input/output and the cluster’s filesystem as transports, but then John told me that he had figured out how to use Erlang’s message-passing/marshaling system. He wrote a Python library for writing fitness evaluators for Erlush, and I employed Erlectricity to do something similar in Ruby. These things both work, on computers where everything is properly installed. (At the time of this writing this does not include the cluster, but we’ll get there.)
• Launching jobs on the cluster: I’ve pretty much finished a fairly straightforward Ruby program to generate Tractor scriptsthat can be spooled to the Tractor job queue. It needs to be more configurable, and I need to work with John to figure out how to collect results from the worker nodes when they finish running.

While it may seem like we’ve been treading water a bit here, I think we’re making real progress, and as our understanding of the best way to put together this infrastructure has evolved, we’ve come a lot closer to our ultimate goal of making running GP experiments on the cluster a one-command operation. Or really, an edit-a-configuration-file-then-run-a-command operation. And we’re really and truly nearly there!

This entry was posted on November 18, 2009 at 11:34 am and is filed under Blag Toasts.