I've been using the flow framework for amazon swf and I want to be able to run priority workflow executions and normal workflow executions. If there are priority tasks, then activities should pick up the priority tasks ahead of normal priority tasks. What is the best way to accomplish this?
I'm thinking that the following might work but I wonder if there's a better/recommended approach.
- I'll define two Activity Workers and two activity lists for the activity. One priority list and one normal list. Each worker will be using the same activity class.
- Both workers will be run on the same host (ec2 instance).
- On the workflow, I'll define two methods: startNormalWorkflow and startHighWorkflow. In the startHighWorkflow method, I can use ActivitySchedulingOptions to put the task on the high priority list.
Problem with this approach is that there is no guarantee that the high priority task is scheduled before normal tasks.
It's a good question, it had me scratching my head for a while.
Of course, there is more than one way to skin this cat and there exists a number of valid solutions. I focused here on the simplest possible that I could conceive of, namely, execution of tasks in order of priority within a single workflow.
The scenario goes as follows: I define one activity worker serving two task lists,
default_tasks
andurgent_tasks
, with a trivial logic:urgent_tasks
list, then pick one from there,default_tasks
The question is how to check if any high priority tasks are pending? CountPendingActivityTasks API comes to the rescue!
I know you use Flow for development. My example is written using
boto.swf.layer2
as Python is so much easier for prototyping - but the idea remains the same and can be extended to a more complex scenario with high and low priority workflow executions.So, to accomplish the above using boto.swf follow these steps:
Export credentials to the environment
Get the code snippets
For convenience, you can fork it from github:
To bootstrap the domain and the workflow:
Decider implementation:
Prioritizing worker implementation:
Run the workflow from three instances of an interactive Python shell
Run the decider:
Start an execution:
Finally, kick off the worker and watch the tasks as they're getting executed: