How to limit number of concurrent workflows running?

Question

The title is pretty much the question. Is there some way to limit the number of concurrent workflows running at any given time?

Some background: I'm using eventarc to dispatch a workflow once a message has been sent to a pubsub topic. The workflow will be used to start some long-running operation (LRO) but for reasons I won't go into, I don't want more than 3 instances of this workflow running at a given time.

Is there some way to do this? - primarily from some type of configuration rather than using another compute resource.

Answer 1

There is no configuration to limit running processes that specifically targets sessions that are executed by a Workflow enabled for concurrent execution. The existing process limit applies to all sessions without differentiating between those from non-concurrent or concurrent enabled Workflows.

Synchronization enables users to limit the parallel execution of certain workflows or templates within a workflow without having to restrict others.

Users can create multiple synchronization configurations in the ConfigMap that can be referred to from a workflow or template within a workflow. Alternatively, users can configure a mutex to prevent concurrent execution of templates or workflows using the same mutex.

Refer to this link for more information.

Answer 2

Summarizing your requirements:

1. Trigger workflow executions with Pub/Sub messages
2. Execute at most 3 workflow executions concurrently
3. Queue up waiting Pub/Sub messages
4. (Unspecified) Do you need messages processed in the order delivered?

There is no out-of-the box capability to achieve this. For fun, below is a solution that doesn't need secondary compute (and therefore is still fully managed).

The key to making this work is likely starting new executions for every message, but waiting in that execution if needed. Workflows does not provide a global concurrency construct, so you'll need to use some external storage, such as Firestore. An algorithm like this could work:

1. Create a callback
2. Push the callback into a FIFO queue
3. Atomically increment a counter (which returns the new value)
4. If the returned value is <= 3, pop the last callback and call it
5. Wait on the callback
6. -- MAIN WORKFLOW HERE --
7. Atomically decrement the counter
8. If the returned value is < 3, pop the last callback and call it

To keep things cleaner, you could put the above steps in a the triggered workflow and the main logic in a separate workflow that is called as needed.

Answer 3

Currently, I don't know of a way of controlling it by a workflow configuration. I did find this related issue in Googles IssueTracker.

In the meanwhile, I suggest utilizing Cloud Tasks to queue workflow executions, which enables you to control the number of concurrent tasks running in parallel by configuring the queue with the desired max_concurrent_dispatches = 3 (under rate-limit).

Unfortunately, I don't know of a way to directly create a new cloud task on a eventarc event. So to do so, I would create a Cloud Function that is triggered by the relevant eventarc/pubsub that adds the new cloud tasks in the pre defined queue.

So the full solution will look like: eventarc -> PubSub -> Cloud-Function -> Cloud-Task (max 3 in parallel) -> Cloud-Workflow Execution.