Speed up your Data processing with OpenMOLE

Mathieu Leclaire // Romain Reuillon // Mark Hammons // Jonathan Passerat-Palmbach

ISCPIF // BioEmergences // FBI // Imperial College London

When you want to run a large scale study of the brain...

But the deadline for MICCAI is getting so close...

You feel like you'll never get those results in a reasonable time

Hopefully your applications are naturally parallel algorithms \o/

Computing power

• Personal computer // 1 to 8 cores
• Computing server // up to 50 cores
• Cluster // up to 200 cores
• Grid // > 2000 cores

What does OpenMOLE do ?

It exposes a workflow formalism for describing an experiment
It transparently delegates computational loads to massively parallel environments

Upscaling

Prototype small
Experiment large

Embed your algorithm as a black box

C
R
C++
Java
Scala
Scilab
Octave
Python
Netlogo
...

Workflow topology

	A Task runs an executable,
	receives and produces Variables.
	Variables navigate from one Task to another by means of Transitions
	eventually doing Loops.

The design of experiments

Design you experiment in OpenMOLE	At runtime

Computing environments

	The same Task can be run localy
	or by taking leverage of the local multicores
	or on a Cluster (PBS for instance)
	or on the Grid.
	...

Environments already supported by OpenMOLE

• multi threading
• remote SSH server
• PBS Cluster
• EGI
• Dirac
• soon: SLURM, SGE, Condor, other cluster batch systems
• ... and later : private and academic clouds

Powered by GridScale

OpenMOLE from the GUI

OpenMOLE from a DSL

val i1 = Prototype[Int]("i1")
val i2 = Prototype[Int]("i2")
val j = Prototype[Int]("j")

val hello = GroovyTask("hello", "j = Model.compute(i1, i2)")

hello addInput i1
hello addInput i2
hello addOutput j
hello addLib "/path/to/model.jar"

val exploration = ExplorationTask(
    "exploration",
    Factor(i1, 0 to 100 by 2 toDomain) x 
    Factor(i2, new UniformIntDistribution take 10)
  )

val ex = exploration -< (hello by 10 on biomed) toExecution        
ex.start

OpenMOLE from a Web server

Can we really use it in the group?

Step 1:

Pack your bags

Static compilation Language specific archive (jar) Yapa (CDE)

Step 2:

Plan your visits

Input variables of your application Parameters to tweak

Step 3:

Choose the travel options

Local execution Cluster Grid

Step 4:

Collect your pictures

Results downloaded to your computer Write the paper :)