Azkaban : Running Cluster of Azkaban Executor Servers

Question

Is it possible for me to run Azkaban Executor Service in a cluster. If one node goes down, then can the other pick up and run the scheduled jobs? I haven't seen any documentation on this kind of set-up.

All that I read somewhere was that we can run only one Executor Service for each web server. But an ideal fault tolerant architecture would be to have web servers and Executor Servers run independently.

Has anyone handled such a situation. Please let me know if you have any pointers..

Thanks, Kranthi

Answer 1

You must add the following lines of configurations into the directory conf of the azkaban-web configuration:

#Multiple Executor
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

I have inserted the following configuration on the DNS based so that they can have a configuration based on multicluster. It is not necessary if you are inside the same network.

insert into executors (id, host, port, active) values (10000,your.dns.com,12321,1);

At this point points take advantage of the useExecutor parameter by setting the ID inserted above or 10000 as a value

Answer 2

Per Azkaban's roadmap, this facility would be available only in 2.7 version. But this is how I plan to handle it for time being.

Multiple executor services can be run behind a load balancer vip. Web and executor are 2 independent services. Web can route the request to any executor service since the job and flow details are persisted in Mysql anyways and hence this set-up would work.

If there is anything that others did it differently, then please let me know.

Thanks, Kranthi