I need more local disk than available to EC2Resources in an AWS Data Pipline. The simplest solution seems to be to create and attach an EBS volume.
I have added EC2:CreateVolume og EC2:AttachVolume policies to both DataPipelineDefaultRole and DataPipelineDefaultResourceRole.
I have also tried setting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for an IAM role with the same permissions in the shell, but alas no luck.
Is there some other permission needed, is it not using the roles it says it uses or is this not possible at all?
The Data Pipeline ShellCommandActivity with has a script uri point to a shell script that executes this command: aws ec2 create-volume --availability-zone eu-west-1b --size 100 --volume-type gp2 --region eu-west-1 --tag-specifications 'ResourceType=volume,Tags=[{Key=purpose,Value=unzip_file}]'
The error I get is: An error occurred (UnauthorizedOperation) when calling the CreateVolume operation: You are not authorized to perform this operation.
I had completely ignored the encrypted authorization message, thinking it was just some internal AWS thing. Your comment made me take a second look, kdgregory. Turns out the reference to the CreateVolume was somewhat of a red herring.
Decrypting the message, I see that it fails with "action":"ec2:CreateTags" meaning it lacks the permission to create tags. I added this permission and it works now.