How can I use docker compose on AWS Batch?

Question

I have a multi-container (docker compose) application. I would like to scale it offline on AWS Batch for processing large volumes of data on S3. My .yml file for docker compose looks something like this:

version: '2'

services:
    container1:
      container_name:
      image:
      ports:

    container2:
      container_name:
      image:
      depends_on: container1
      ports:

I, unfortunately, cannot find any examples or tutorials online dealing with such a case. Can anyone help me understand how should I approach this problem?

Answer 1

Date: 11-06-2018:

Currently, it is not possible, if it is a multi-container docker compose, you can't. There's no option for a docker-compose up instruction, just a single docker image is allowed. The definition would be like next:

"Properties" : {
    "Type" : "container",
    "Parameters" : {
      "net": "host"
    },
    "ContainerProperties" : {
      "Command" : [ "command1" ],
      "Memory" : 2000,
      "Environment": [
          {
            "Name" : "NAME",
            "Value" : "true"
          }
      ],
      "Vcpus" : 2,
      "Image" : "redis:latest"
    },
    "JobDefinitionName" : "job-1",
    "RetryStrategy" : {
        "Attempts" : 1
    }
}

In fact in the official docs is specified container as the only valid value for type:

• https://docs.aws.amazon.com/batch/latest/userguide/job_definition_parameters.html#type

Type type When you register a job definition, you specify the type of job. At this time, only container jobs are supported.

Type: String

Valid values: container

Required: Yes

---

For individual containers there's a cli tool to transform docker-compose to a job definition but actually, that's something you can do manually:

• https://github.com/turnerlabs/compose-to-batch

Answer 2

I'm not sure what the capabilities were when this was posted but I recently had a project for this issue.

{
  "jobDefinitionName": "batch-job-definition",
  "containerProperties": {
    "image": "public.ecr.aws/docker/library/docker:20-dind",
    "command": ["docker-compose","-f", "/mnt/efs/docker-compose.yml","up"],
    "volumes": [
      {
        "efsVolumeConfiguration": {
          "fileSystemId": "fs-XXXXXXXXXXXXXXXXX",
          "rootDirectory": "/",
          "transitEncryption": "ENABLED",
          "authorizationConfig": {
            "accessPointId": "fsap-XXXXXXXXXXXXXXXXX",
            "iam": "ENABLED"
          }
        }
      }
    ],
    "mountPoints": [
      {
        "containerPath": "/mnt/efs",
        "readOnly": false,
        "sourceVolume": "efs-volume"
      }
    ],
    "user": "root",
    "networkConfiguration": {
      "assignPublicIp": "ENABLED"
    }
  },
  "platformCapabilities": [
    "EC2"
  ],
  "containerOrchestrationType": "ECS"
}