Context: We are receiving files which are PGP encrypted , which needs to be decrypted in AWS Glue using pyhon-gnupg module
Issue: AWSGlue with PySpark is able to decrypt files upto 2GB files but we are receiving large files up to 50GB and 1TB in S3 bucket, AWSGlue throws error like " command failed with exit code 10 "
How do we decrypt large files using AWSGlue PySpark and python-gnupg
Current Code:
import gnupg
gpg = gnupg.GPG()
gpg.import_keys(private_key)
with self._fs.open(s3FileSourcePath, 'rb') as fd:
decryptedFile = gpg.decrypt_file(fd, passphrase=secret_dict['PassPhrase'])