Cluster Support for Genetic Programming

It computes factorial. Here it is:

[{exec,pop}, [[[{exec,do_star_times},{integer,yank}],[[-9,{integer,yankdup}]]], [{boolean,stackdepth}, [[{integer,flush},{exec,b},{exec,k}], {exec,y}, [[[[{boolean,rot},{exec,do_star_times}]]], {integer,divide}, {integer,yank}]], [[[[{exec,do_star_count},{integer,lessthan}]]]]]], {integer,equal}, [{exec,do_star_times}, [[{exec,s},{exec,do_star_times}],[{integer,max},{boolean,yankdup}],{exec,c}], {exec,k}, {integer,multiply}]]

And the same program after simplification (by hand):

[{exec,do_star_times}, [[{exec,do_star_times}],{exec,c},[[],{exec,c}]], {exec,k}, {integer,multiply}]

And again in nicer, lispy, Push3 syntax

(EXEC.DO*TIMES ((EXEC.DO*TIMES) EXEC.C (() EXEC.C)) EXEC.K INTEGER.*)

I was pretty excited to see C combinators in there, perhaps they’re a worthwhile addition to the language.. No worries if you can’t figure out what’s going on here, I’ve single stepped through it with Erlush in debugger mode and still can’t say for sure that I know how it works. I did, however, discover that it’s exploiting a bug in my implementation of EXEC.DO*TIMES — which is discouraging and exciting at the same time.

More to come.

John

UPDATE: I fixed the EXEC.DO*TIMES bug and reran the factorial evolver, and got (after simplification and Push3ifying)

(EXEC.STACKDEPTH INTEGER.DUP INTEGER./ EXEC.DO*RANGE INTEGER.*)

Which is much clearer. The first three instructions just put a 1 on the stack on top of input and then EXEC.DO*RANGE INTEGER.* iterates from the input value down to 1 multiplying the loop counter and the value beneath it each time.

I talked to Josiah earlier today and got him to install Xvfb (X Virtual FrameBuffer) on the cluster; so now you can run Processing and other graphical applications on there. Xvfb emulates a graphical display, so as far as your programs are concerned they’re running in a full fledged graphical environment. All you have to do is launch your application with my “headless” script. Headless simply creates a virtual display with Xvfb, and tells your program to use it instead of the default display.

To use headless:
Download the script and copy it to your home directory on the cluster:

scp -p -r headless YourUserName@fly.hampshire.edu:~/

Now say your graphical program is called doawesome and takes three arguments. Instead of telling tractor or multiquery to issue the command:
./doawesome 1 33 7
You tell it to use the headless script instead:
./headless ./doawesome 1 33 7

One more tip for processing users:
When you’re ready to run your sketch on the cluster, open it in processing and click File->Export application, then select Linux and hit export. This should create a directory called application.linux inside your sketch’s directory, and within application.linux a shell script that will launch your sketch. To run it on the cluster just copy this application.linux directory to your home directory:

scp -p -r YourSketch/application.linux YourUserName@fly.hampshire.edu:~/

And issue jobs for the headless script to launch the shell script inside of ~/application.linux

multiquery './headless ~/application.linux/doawesome'

I wrote a little script that uses ganglia (The cluster monitoring toolkit which was already installed on the cluster) to generate lists of nodes sorted in order of lowest cpu usage. It can also sort by memory free and several other metrics.

Example usage (Get 10 nodes, sorted by lowest cpu usage):

[jms07@fly ~]$ ./gquery -c 10
compute-0-2
compute-0-3
compute-0-4
compute-0-5
compute-0-6
compute-0-7
compute-0-8
compute-0-9
compute-1-5
compute-1-10


Changelog
Tarball
Git: git://anomos.info/~john/ClusterSupport

-John

I’ve made the source code for my Erlang implementation of Push available at anomos.info/~john/. I’m currently calling it PushAgner, this is a terrible name (Erlang is named after the inventor of queuing theory, Agner Krarup Erlang), please help me come up with a better one.
EDIT: I’ve renamed it to Erlush, which was another working title I had used, and was suggested by Lee.

If you have git installed, you can get the source by running:
$ git clone git://anomos.info/~john/erlush
Following which you can get any updates by running:
$ git pull

If you don’t have git, you can just grab a tarball of the latest source.

I haven’t written any documentation yet. If you really want to get this running it’s probably best to contact me directly.

– John