How to create a fan-out DeferredTask in Google App Engine - JAVA

Question

I have a Java App Engine project and I am using DeferredTasks for push queues.

/** A hypothetical expensive operation we want to defer on a background task. */
public static class ExpensiveOperation implements DeferredTask {

  @Override
  public void run() {
    System.out.println("Doing an expensive operation...");
    // expensive operation to be backgrounded goes here
  }
}

I want to be able to create multiple shards of a DeferredTask to be able to have more through-put. Basically, I want to run one DeferredTask that then runs many more DeferredTasks (up to 1,000 of them). Essentially a fan-out task. How can I do that?

One issue is that when creating tasks you need to specify the name of them in the queue.yaml file. But if I want to have 1,000 tasks, do I really need to specify 1,000 of them in that file? It would get very tedious to write out "task-1", "task-2", etc.

Is there a better way to do this?

Answer 1

This is usually done by specifying a shard parameter for each task and reusing the same queue. As noted in your example, the entire java object is serialized with DeferredTask. So you can simply pass in any values you want in a constructor. E.g.

public static class ShardedOperation implements DeferredTask {
  private final int shard;
  public ShardedOperation(int shard) {
    this.shard = shard;
  }
}
...
@Override
public void run() {
  System.out.println("Fanning out an expensive operation...");
  Queue queue = QueueFactory.getDefaultQueue();
  for (int i = 0; i < 1000; ++i) {
    queue.add(TaskOptions.Builder.withPayload(new ShardedOperation(i)));
  }
}

This matches the section you linked to https://cloud.google.com/appengine/docs/standard/java/taskqueue/push/creating-tasks#using_the_instead_of_a_worker_service where the default queue is used.