Mesos Framework

Question

I want to distribute Docker containers on a cluster by using java and define dependencies, so that when one container is finished I can parse the produced output in my java code. There are many mesos frameworks that can achieve that and I'm not sure which one to take: Marathon, Singularity, Chronos, Aurora.

What I have learned so far:

Marathon has a really nice java client api but is used for long term tasks (don't know if that is a problem, because my tasks will not run that long) and I dont think that I can define that kind of dependency. If I would use marathon I would have to poll for the status of the app. Chronos has no java api (At least I could not find any). So I can choose between Aurora and Singularity.

Can anyone help and recommend me one of them for my use case.

Answer 1

Apache Aurora added Docker support in February's 0.7.0 release, and can manage both long-running services as well as cron using the same scheduler. It's also written in Java.

There's a great tech talk from the developer who contributed this feature about how they schedule Docker containers using Aurora. Although the examples provided in the talk are for long-running services, there's just a minor change your job configuration to specify a service vs cron.

Answer 2

Chronos (like Marathon) is written in Scala, so it should be easy to integrate with any JVM-based app. Even better, they both (probably all 4 actually) have REST APIs, so you can just curl your actions/queries, regardless of each app/service's implementation language.

For your use case, you'll probably prefer Chronos over Marathon, since you expect your tasks to complete (and not be restarted). Marathon is intended for long-running (marathon, running.. get it?) tasks, so it will restart a failed/completed task. Chronos is like a distributed cron with dependencies, so you can schedule your jobs, and then add dependent jobs to start upon completion of previous jobs, building arbitrary DAGs of jobs.

I'll let others chime in about Aurora and Singularity. I think they both support cron-like jobs.

Answer 3

Chronos is good for that. You can create dependency DAGs in Chronos. However, it doesn't provide you mechanisms to pass the parameters from one job to other. You have to serialize and deserialize the stuff in DB or shared storage across the cluster.