Open MOdeL Experiment

Mathieu Leclaire
Romain Reuillon
Jonathan Passerat-Palmbach



The Complex system modelisation




These methods are time consuming

Naturally parallel algorithms permit to leverage parallelism

Chromosome structuring


C++ model
2 days per simulation for 1600 simulations
= 8.5 years CPU time

The SimTRAP project


Netlogo model
5 mins per simulation for 100,000 simulations
= 1 year CPU time

The Simpop project


Scala model
2 secs per simulation for
500,000,000 simulations
= 30 years CPU time

The Bioemergences project


C model - Portal access
Daily production
10,000 hours / day

OpenMOLE in 3 steps


Embed your model as a black box

Zero deployment approach

  • Code is embedded with runtime
  • Ships to remote environment
  • No prior knowledge of remote environment needed
  • No installation required on any machine

Naturally parallel formalism to design experiments



What OpenMOLE does



Data parallelism

What OpenMOLE does not



Parallelisation by message / Task Parallelism
Example: Spark, MPI, ...

Powered by EGI

(and others)

So hard to use...

Automatic data transfers
and
Replica management


A workflow


val i   = Val[Double]        
val res = Val[Double]        

val exploration = ExplorationTask ( i in (0.0 to 10.0 by 1.0) )

val model = 
  ScalaTask ("val res = i * 2") set (
    inputs  += i,
    outputs += (i, res)
  )
  
val env = LocalEnvironment(5)

val ex = exploration -< (model on env) start
        

Web Application

Console interface


Towards a multi-user web platform


Towards graphical workflows


Useful Links

Documentation www.openmole.org
Mailing-list list.openmole.org
Development version next.openmole.org
Source code github.com/openmole
Market place github.com/openmole-market

Hands-on

openmole-market
Full examples available

Setup your environment

Download openmole: next.openmole.org
Don't close the tab you'll need it for the documentation ;)

java -version
java version "1.7.0_75"
OpenJDK Runtime Environment (IcedTea 2.5.4) (7u75-2.5.4-2)
OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)
        

~/.openmole

Stores your preferences
Maintains a table of file locations
Can grow rapidly in number of files
=> move it to a folder without quota restrictions

Launch OpenMOLE

Command-line console: openmole -c
Web UI: openmole

1st example

Pi Monte-Carlo calculation www.doc.ic.ac.uk/~jpassera/openmole-tutorial-material

1st example

DoE to achieve

1st example

Add variables declaration
Define exploration
Define environment

1st example

Remote environment?
  • PBS
  • Slurm
  • Condor
  • EGI
  • ...


Define Authentication

Extra notions

Hooks

Tasks are mute
Hooks extract content from the dataflow

exploration -< (model on env) >- (average hook ToStringHook())
        

Grouping

Short jobs don't cope well with distributed computing
Group multiple tasks in the same batch job

exploration -< (model on env by 10) >- (average hook ToStringHook())
        

Errors


exploration -< (model on env by 10) >- (average hook ToStringHook())
errors(env)
        

logger.level("FINE")
        

2nd example

Tree leaves classifier www.doc.ic.ac.uk/~jpassera/openmole-tutorial-material

Output


Random Forest


Design
of
Experiments

Packaging an application

Applications have dependencies:
  • Shared libraries
  • Packages
  • Low level system calls
  • Environment variables
  • ...

Capture these dependencies and transfer along with the application

CARE

http://reproducible.io/
  • Run against your application
  • Transfer archive
  • Re-execute
  • Change the parameters
  • care -r ~ -o archive.bin python script.py 42
    Only from Linux to Linux

    2nd example

    • Add variables declaration
    • Define exploration
    • Define environment
    • Collect results as CSV

    Results visualisation?

    • Add another task
    • Runnning locally
    • Example: Python + Matplotlib scatter plot

    Going further

    • Sources
    • Advanced DoE
    • Desktop Grid
    • Plugins
    • Strainer Pattern

    Thanks!

    romain.reuillon@iscpif.fr
    mathieu.leclaire@iscpif.fr
    j.passerat-palmbach@imperial.ac.uk