This is the first MR which will eventually deliver the full jobs API, but I have decided to take stock and start reviewing and merging this MR before things start to grow too much and it's unreviewable. Furthermore I'm off this afternoon so I wanted to make sure we had something concrete to start working with and collect feedback on.

Here is the gist: even though it might seem this MR is not adding much, it's paving the way to have an API to track the status of each individual job, as well as cleaning up some of the boilerplate we have in the system.

The main idea is that we now have a new capability called MonadJobStatus, which currently doesn't have any typeclass methods but that it has the purpose of tracking (no pun intended) the effect of reporting job statuses using the inner monad m.

I have refactored ever so slightly the existing jobs API to be more polymorphic, and now, in the callback of serveJobsAPI, instead of be given a Logger w, we are now given a JobHandle: the intuition would be similar to a file handle or a DB connection handler, it's an opaque reference from which you can access an API, in this case, the API to track a job's status.

The API we have so far is:

getLatestJobStatus :: MonadJobStatus m => JobHandle m (JobEventType m) -> m (Maybe (JobEventType m))

updateJobProgress :: (Monoid (JobEventType m), MonadJobStatus m)
                  => JobHandle m (JobEventType m)
                  -- ^ The handle that uniquely identifies this job.
                  -> (JobEventType m -> JobEventType m)
                  -- ^ A /pure/ function to update the 'JobEventType'. The input
                  -- is the /latest/ event, i.e. the current progress status. If
                  -- this is the first time we report progress and therefore there
                  -- is no previous progress status, this function will be applied
                  -- over 'mempty', thus the 'Monoid' constraint.
                  -> m ()

With the first function we can get the latest JobEventType (i.e. a JobLog, concretely), whereas updateJobProgress is currently a fairly general function that can be used to update a JobLog, as it essentially takes a JobLog -> JobLog as input.

Further refinements of this MR will cut more boilerplate by adding methods to the MonadJobStatus typeclass which can be implemented, concretely, using things like jobLogSuccess, to avoid endless repetition.

On the topic, I feel there is a design decision we have to make, which perhaps @anoe and @cgenie can help out with; currently our jobs api is quite dynamic, in the sense that we don't track, at the type level, the number of steps for each job, and if, say, calling jobLogSuccess or similar does the correct thing: in theory it's possible to screw up and end up with a _scst_remaining which is greater or even lower than 0, if we didn't do the math correctly when reporting things. In theory I think we might be able to track this at the type level, but it might make this whole API a bit heavy and intimidating to use, but nevertheless I guess that's an option we could explore. Let me know what is your preference; I suspect that for an initial version of this API we don't need such complication.