Worflows for distributed computing

Mathieu Leclaire
Jonathan Passerat-Palmbach
Romain Reuillon

Context

Complex-system community
Various scientific fields
Thematician PhDs (Geographers, Biologist, Scociologist...) with no technical support
No standard practices, language, plateform..
They use naturally parallel methods daily on their laptop:
  • Data reconstruction
  • Parameter estimation
  • Sensitivity analysis
  • Optimisation
  • Replication
  • ...
Execution on the same program with different parameters and/or datasets.
How to bring Distributed Computing to theses researchers
Prototype Small, Scale for Free
OpenMOLE articulate 3 orthogonal concepts

Tied by an expressive workflow formalism for distributed computing.

1 - Model?



Stuff that you can launch, taking inputs and producing outputs

Zero deployment approach

User code is automatically deployed at runtime
No installation is required on the execution environment
No prior knowledge of the execution environment is needed
Works with almost any language / plateform running on Linux

Packaging (non JVM) application with Care

Applications have dependencies:
  • Shared libraries
  • Packages (Python, R, ...)
  • Environment variables
  • ...

Capture these dependencies and transfer along with the application from Linux to Linux
2 - Method?

Map/reduce


Map/reduce

Grid Search

Random sampling

Latin Hypercube

Parallel data processing

...

Master/slave

Example: genetic algorithm

3 - Execution environment?

Today

Multi-thread
Delegation through SSH
PBS (on ssh)
SLURM (on ssh)
Condor (on ssh)
SGE (on ssh)
OAR (on ssh)
DIRAC
Adhoc Desktop Grid

Tomorow

Commercial cloud providers
Academic cloud

Volonteer computing?
Combination of environments?
Next docker-based computing plateform?

Transparent access

No preliminary step

Access as the user would do it

With user credential

Automatic data transfers
(+ Replica management)

Files and folders transfers are handled transparently by OpenMOLE

And now: examples!

The terminology



A workflow

val inputFile   = Val[File] 
val fileName    = Val[String]
val outputFile  = Val[File]        

val process = 
  ScalaTask ("""
    val outputFile = newFile()
    outputFile << inputFile.content.toUpperCase""") set (
    inputs  += inputFile,
    outputs += outputFile,
    (inputs, outputs) += fileName
  )

val exploration = 
  ExplorationTask ( 
    inputFile in (workDirectory / "files") withName fileName 
  )
  
val copy = CopyFileHook(outputFile, workDirectory / "results/${fileName}")

exploration -< (process hook copy)

val inputFile   = Val[File] 
val fileName    = Val[String]
val outputFile  = Val[File]        

val process = 
  ScalaTask ("""
    val outputFile = newFile()
    outputFile < inputFile.content.toUpperCase""") set (
    inputs  += inputFile,
    outputs += outputFile,
    (inputs, outputs) += fileName
  )

val exploration = 
  ExplorationTask ( 
    inputFile in (workDirectory / "files") withName fileName 
  )
  
val copy = CopyFileHook(outputFile, workDirectory / "results/${fileName}")

val biomed = EGIEnvironment("biomed") // Add requirements here

exploration -< (process hook copy on biomed by 2)

Native code

care -o hello.tgz.bin python hello.py 42 test.txt

val arg = Val[Int]
val output = Val[File]

val pythonTask =
  CARETask(
      workDirectory / "hello.tgz.bin",
      "python hello.py ${arg} output.txt") set (
    inputs += arg,
    outputFiles += ("output.txt", output),
    outputs += arg
  )

val exploration = ExplorationTask(arg in (0 to 10))

val copy = CopyFileHook(output, workDirectory / "hello${arg}.txt")

exploration -< (pythonTask hook copy)
    
 val evolution =
  SteadyStateEvolution(
    algorithm =
      NSGA2(
        mu = 200,
        genome =
          Seq(
            rMax in (2.0, 50000.0),
            distanceDecay in (0.0, 4.0),
            pCreation in (0.0, 0.01),
            pDiffusion in (0.0, 0.01),
            innovationImpact in (0.0, 2.0),
            innovationLife in (1.0, 4001.0)
          ),
        objectives = Seq(ksValue, deltaPop, deltaTime)
      ),
    evaluation = evaluateModel,
    termination = 15 minutes
  )

val island = IslandEvolution(evolution, parallelism = 1000, termination = 200000)

val savePopulation = SavePopulationHook(island, workDirectory / "populations")

val grid = EGIEnvironment("vo.complex-systems.eu")

(island on grid hook savePopulation)

Useful Links

Documentation www.openmole.org
Development version next.openmole.org
Source code github.com/openmole
Market place github.com/openmole-market

Thanks!

romain.reuillon@iscpif.fr
mathieu.leclaire@iscpif.fr
j.passerat-palmbach@imperial.ac.uk