Node assignment has to be done based on priority value but not on Job

Question

I have multiple pipeline jobs triggered parallely at scheduled time. Each job consists of multiple stages to be created dynamically based on the list of elements. And each stage will get assigned one node from Node Label and execute some Linux commands on it, based on the availability.

How to prioritize the Node assignment (Common Node label for all jobs) to each stage for different jobs based on the availability and priority given for a Job.

Lets take jobs Job1, Job2, Job3, Job4 triggers at 12 PM and have 100 Nodes with common label "Execution_Pool" and each Job stage were allocated node dynamically based on list to do some executions in each stage. Now if I trigger another job (job5) at 5 PM by setting priority as highest, then this Job5 should take the freed-up node from Execution_Pool rather than other jobs which are triggered at 12 PM.

I have tried to use Priority Sorter plugin, but I understood that it is meant for only jobs but not on nodes allocation for each stage at runtime.

I'm expecting these nodes will be allocated by looking into the priority value for each job given.

Lets say less value has higher priority.

Answer 1

From your description, it's not entirely clear what you are trying to achieve, so I won't add any code as it can solve a problem entirely different to yours.

I don't know of way to interfere with (or give hints to) Jenkins scheduler except for adding or removing labels in your jobs. If stickiness to nodes is what you're after, your Job1 can theoretically get a node, add inuse_job1 label to that node, and later schedule stages on that label instead of the usual label (don't forget to clear that label in the end of the job).

If you want an "emergency" job to come before (and at expense of) any other job, the way to do it is to start by going over all the nodes, adding a label emergency to the node while renaming every existing label (e.g. by adding a suffix), then schedule your emergency job's stages on that emergency label, and in the end of the job clear the emergency label and the suffix at the other labels. With the renamed labels, all the jobs that seek a node with a regular label will be waiting (as no nodes with that label are there), while the emergency job will have lots of available resources.

Example:

Assume you have two nodes, X and Y, both have the label pool.

Assume a job takes exactly 6 hours.

Job A starts at 12:00, schedules on pool label, gets node X and starts executing. Job B starts at 13:00, schedules on pool label, gets node Y and starts executing. Job C starts at 14:00, schedules on pool label but does not get any node, so it waits.

The status of pool label after 14:00: available nodes - 2, busy nodes - 2, waiting jobs - 1 (C).

At 17:00, emergency job D starts executing.

First thing, it changes labels on nodes: both X and Y are now labeled with emergency and pool_emergency (none are labeled pool). It then schedules execution on emergency label.

The status of pool label after 17:00: available nodes - 0, busy nodes - 0, waiting jobs - 1 (C).

The status of emergency label after 17:00: available nodes - 2, busy nodes - 2, waiting jobs - 1 (D).

At 18:00, job A is finished, and node X is free. This node (labeled emergency and pool_emergency) goes to Job D that is waiting for emergency label.

At 19:00, job B is finished, and node Y is free. This node does nothing; in the meantime job C is still waiting for pool label.

At 24:00, job D is finished. Nodes X and Y have the emergency label removed and pool_emergency returned to pool. Now job C gets a node and starts executing.

In this way, we made a later (emergency) job D execute before an earlier (non-emergency) job C.

This provides a way to have two levels, "normal" and "emergency". If there is a need, you can expand it above two labels by carefully crafting suffixes in your labels.