Dataproc: Can user create workers of different instance types?

Question

scenario: master: x1 machine type workers: x2-machine type, x3-machine type.

For the above scenario: AWS EMR instance fleet allows users to create different worker instance types. From dataproc console, I noticed options is for only, N-worker types(all worker types of same type). Does Dataproc HDI also support this option of creating different instance types for worker?

Answer 1

Dataproc supports 2 types of worker group: (primary) workers and secondary workers. But they must have the same machine type as of Q2 2022, there is only one flag --worker-machine-type for it.

Note that it does allow secondary workers to have different disk type, disk size, accelerator, etc. See this doc for more details.