I am using AWS Fargate to run containerized Batch jobs. Before submitting jobs, I need to:
- create and push a container to Amazon Elastic Container Registry
- create a Job definition (referencing the container from the previous step)
- create a Compute environment
- create a Job queue
The Compute environment has a Compute resources section, where the number of Maximum vCPUs are specified. I set it to 32.
The fact that I can set this value to 32 seems to be in conflict with what I read about AWS Fargate service quotas. Based on the table in the documentation:
I should not be able to use 32 vCPUs.
I have a job queue, job-queue-B, where I would like to submit as many jobs as possible at the same time. The container of each job is allocated 0.25 vCPUs.
While running jobs in job-queue-B, I must not block processing in job-queue-A. Occasionally, a job is submitted to job-queue-A, with a container having 4 vCPUs, in a compute environment with maximum 1024 vCPUs. This job must be started immediately, and it must not have to wait for any processes in job-queue-B.
I do not understand how Compute environment vCPUs can be set so much higher than the AWS Fargate service quota limit quoted above.
That being said, my main question is:
How many jobs can I submit to job-queue-B at the same time?
A seemingly reasonable approach:
Just divide the Maximum vCPU allocation of the compute environment with the vCPUs required for each job. 32/0.25=128 jobs can run at the same time.
Since the number of vCPUs in the compute environment was set rather arbitrarily, this approach does not answer the question of how many jobs I can possibly run, if I am also able to adjust number of vCPUs in the used compute environment (I am able to).
Another seemingly reasonable approach, with vastly different end result:
Since job-queue-A jobs are running in a container with 4 vCPUs allocated, subtract this from the quota of 6 vCPUs. Divide by the number of vCPUs job-queue-B jobs need. (6-4)/0.25=8 jobs can run at the same time.