ruffus

in looking into Python-based workflow systems to be used in highly parallelizable material science applications that will run on Titan, we came across ruffus

there is a lot to wrap one's head around in this project so first i am just going to do a simple little experiment to see how it will coordinate activities on my local machine

after that i think i should figure out how to simulate getting it to run on multiple nodes. without access to a supercomputer, I am not sure how I'll do this.

since we'll be interfacing with SGE and ruffus supposedly offers support for SGE, I think we'll be in good shape. i just need to understand how it works instead of expecting magic

i think i'll do a fractal image generation as a problem to solve and test this out

I'll probably borrow almost exclusively from James Tauber's work on Littlewood

The first thing I am going to do is fork littlewood and then create a parallel branch.

Here is where I'll add my work to extend things to have parallelization via ruffus.

Starting parallel.py the first thing I'll do is pull in parts of roots.py and heatmap.py and pull every thing into discrete functions.

There is parity now (verified by running) but I did remove the writing roots to a file and instead keeping them in memory. My thinking is that this will make it easier to parallelize the roots generation. We'll see.

I have replaced the normal sequential run of the top level roots_for_degree and heatmap functions with making them tasks and running them in the ruffus.pipeline_run.

Still no parallelization. That's next. This just simulates exactly how the script ran before.

Interesting thing to note, that my previous runs of 16 degree, size 200 fractals, took approx. 33-36 seconds. Now replaced to run in the pipeline they take 45 seconds. Seems like a good bit of overhead, but perhaps it isn't linear, and will more than make up for running roots in parallel.

Finished the conversion, running with multiple processes certain speeds up the roots calculation a bit, but for my Mac Book Air, with only 2 cores, it's hard to see that much of a difference.

I wonder what it would do on larger machines with more CPU and more cores. Adjust how many processes to run in parallel by changing the multiprocessing argument at the bottom of the file.

The next phase of this research is figuring out how ruffus interfaces and spawns work off to SGE.

Picking up on this adventure, I am going to try to use SAGA-python as an abstraction layer of a different distributed execution environments to run external processes.

I had originally thought SAGA-python was on googlecode which I was a bit dismayed by because I already see a change that I want to contribute back in the code. So happy to learn it's on Github.

First step is going to be pulling out:

Into a stand alone script and then replace the internals of that function with a SAGA execution of that new script.

Working with SAGA-python is very hard.

Finally got remote execution via SAGA/ssh with SFTP transfers in a ruffus job, but it was painful.

Also, it seems very flakey.

Flakey in the since that i am getting IO size mismatch errors that are non-deterministic. My connection is via localhost so I would not expect any network issues.

Ok, the flakiness was all my doing.

I had copy and pasted from some example code and didn't correct for my particular use case.

I do wonder if these remote notes will be prepared with a run time environment, or will I be responsible for bootstrapping all of them.

Another thing to sort out is how to choose a node.

With a PBS cluster, you can queue the task, so presumedly the cluster is choosing the node for you.

I guess the point of "grid" computing is that you don't pick a node, but rather you address a cluster.

I do wonder how you bootstrap nodes in your cluster. Surely we don't have to bootstrap an execution environment every time.

I'd like for it to be a split between something like providing an image with a virtualenv prepped with numpy and any other lib that might be needed, but then the code that needs to execute we copy over at run time like:

https://github.com/eldarion/littlewood/blob/parallel/parallel.py#L74

If this is the case, this perhaps when I do this it will be fine to be hard coded.

I plan on going through the SAGA-python interface, but reading about SGE makes me wonder just how this bootstrapping is done.

Looking at StarCluster by MIT to help provision clusters on EC2 for testing.

This screencast captures what looks to be a pretty awesome tool for launching SGE clusters on EC2

So I got a 10-node cluster fired up with little problem.

Now it's just getting the SGE plugin in SAGA-python working. So far just all kinds of SSH connection issues.