GridScale: a journey from Object-Oriented to Functional Programming
Mathieu Leclaire
Jonathan Passerat-Palmbach
Romain Reuillon
Context
Need a library to submit jobs
How does it look?
val slurmService = new SLURMJobService
with SSHPrivateKeyAuthentication {
def host: String = "server.domain"
def user = "user"
def password = "password"
def privateKey = new File("/path/to/key/file")
}
val description = new SLURMJobDescription {
def executable = "/bin/echo"
def arguments = "success > test_success.txt"
def workDirectory = "/home/user"
def walltime = 10 minutes
}
val j = slurmService.submit(description)
Design
Cake pattern
/ Mix-ins
trait SLURMJobService extends JobService
with SSHHost
with SSHStorage
with BashShell
Problems
trait SLURMJobService extends JobService
with SSHHost
with SSHStorage
with SSHSimpleConnection
with SSHCachedConnection
with BashShell
- Which Connection is used?
- Implementation leaks in type
I just want to submit a job!
f: jd => js => job
def submit[D, S](desc: D, jobService: S)
Functional patterns in GridScale
Type classes
trait SSHAuthentication[T] {
def authenticate(...): ...
}
case class PrivateKey(...)
case class LoginPassword(...)
implicit def privateKeySSH = new SSHAuthentication[PrivateKey] {
def authenticate(...) = ...
}
def submit[A](a: A, d: Description)
(implicit c: SSHAuthentication[T])
Reader Monad
- Dependency injection => connection
- Pollutes call stack
- Side effects appear explicitly all the way down the call stack
Free Monad
- Focus on API
- Highly composable DSLs
- Push side-effects to the boundaries of the program
- Change behaviour using different interpreters
- Overall pattern contains still a lot of boilerplate (use Freek at the very least)
github.com/ProjectSeptemberInc/freek
FreeMonad - concrete example
- You've seen the logger already :)
- Pseudo-Random Number Generation
A Random FreeMonad...
object Random {
def interpreter(random: util.Random) = new Interpreter[Id] {
def interpret[_] = {
case nextDouble() => Right(random.nextDouble)
case nextInt(n) => Right(random.nextInt(n))
case shuffle(s) => Right(random.shuffle(s))
}
}
def interpreter(seed: Long): Interpreter[Id] =
interpreter(new util.Random(seed))
}
@dsl trait Random[M[_]] {
def nextDouble: M[Double]
def nextInt(n: Int): M[Int]
def shuffle[A](s: Seq[A]): M[Seq[A]]
}
How will it look? - WIP
def interpreter(client: util.Either[ConnectionError, SSHClient]) =
new Interpreter[Id] {
lazy val sFTPClient = client.map(_.newSFTPClient)
def interpret[_] = {
case execute(s) ⇒ for {
c ← client
r ← SSHClient.exec(c)(s).toEither.leftMap(t ⇒ ExecutionError(t))
} yield r
case fileExists(path) ⇒
sFTPClient.map(_.exists(path))
case readFile(path, f) ⇒
sFTPClient.map { c ⇒
val is = c.readAheadFileInputStream(path)
try f(is) finally is.close
}
}
}
@dsl trait SSH[M[_]] {
def execute(s: String): M[ExecutionResult]
def fileExists(path: String): M[Boolean]
def readFile[T](path: String, f: java.io.InputStream ⇒ T): M[T]
}
Usage (merge with UUID)
import freek._
import cats.implicits._
import freedsl.random._
import freedsl.util._
val c = freedsl.dsl.merge(Util, SSH, Random)
import c._
def randomData[M[_]](implicit randomM: Random[M]) = randomM.shuffle(Seq(1, 2, 2, 3, 3, 3))
def job(data: String) =
SSHJobDescription(
command = s"echo -n $data",
workDirectory = "/tmp/")
val prg =
for {
sData ← randomData[M]
jobId ← submit[M](job(sData.mkString(", ")))
_ ← implicitly[Util[M]].sleep(2 second)
s ← state[M](jobId)
out ← stdOut[M](jobId)
} yield s"""Job status is $s, stdout is "$out"."""
Summary
- Object didn't work for us
- Initial design impacts future enhancements
- Learnt it the hard way...
- Functional patterns provide (more?) powerful abstractions
So what's next?
- Massive refactor :)
- API will change (road to 2.0)
- Come learn better design with us!
- Check the issues list to contribute ;)