Receiving "Unexpected field: 'expiration'" error while calling ClientEncryption.createDataKey() in Node/Mongo/AWS environment with explicit encryption

Question

Background and Problem

My Node v18 server application is deployed on AWS Elastic Beanstalk on an EC2 instance running Amazon Linux 2023. I've set up my .ebextensions to set up the Mongo Enterprise repository, adapted from here, and I've installed the mongo-enterprise-cryptd binary from that repo in order to support the mongo-client-encryption node library that my app server needs to validate the fields I'm explicitly encrypting and implicitly decrypting.

My initialization code sets the KMS provider to AWS, and sets its value to a blank object {}, which prompts AWS to look for an IAM role that injects the environment values at runtime, as described in the "Important" section just below here.

Since Mongo has not yet released the core C encryption library libmongocrypt as a package for Amazon Linux 2023 (afaik as of Dec 2023; Here's the MongoDB JIRA ticket I created), I've compiled it from source on my EC2 instance. After this I installed and started the mongocryptd process, then installed the mongo-client-encryption node driver. This all appears to work as my app does not complain about any packages being missing.

During my application's initialization of the encryption keystore database, I call ClientEncryption.createDataKey() with the KMS options described in the docs, and I one of my catch blocks throws this error:

 /var/app/current/my-server/src/encryption.ts:167
        throw new Error('Unable to create Data Encryption Key (DEK)::' + err);
              ^
 Error: Unable to create Data Encryption Key (DEK)::MongoCryptError: Unexpected field: 'expiration'
    at call (/var/app/current/my-server/src/encryption.ts:167:15)
    at tryCatch (/var/app/current/my-server/src/encryption.ts:2:1)
    at Generator._invoke (/var/app/current/my-server/src/encryption.ts:2:1)
    at Generator.throw (/var/app/current/my-server/src/encryption.ts:2:1)
    at asyncGeneratorStep (/var/app/current/my-server/src/encryption.ts:2:1)
    at _throw (/var/app/current/my-server/src/encryption.ts:2:1)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

This part: MongoCryptError: Unexpected field: 'expiration' is the salient bit.

I do not define an encryption schema at any point in my code since I'm using explicit encryption. I do not provide a field in the KMS or connection options that is called 'expiration'. I do not have a field anywhere in my mongoose schemas that is called expiration.

Questions:

1. Is Amazon injecting this field when the Mongo node driver fetches the KMS credentials?
2. If not, where is this field coming from?
3. How can I resolve this or bypass checks to fields I do not care about?

Attempted:

• Attempted to call the ClientEncryption.createDataKey() method with "appropriate" configuration options

Expected:

• Data encryption key to be created and code to continue executing

Actual:

• Received an error referencing a field key that does not exist anywhere in my code base:

MongoCryptError: Unexpected field: 'expiration'

Answer 1

Cause: Missing Environment Variables in AWS Elastic Beanstalk (EB)

Background

The cryptic error: MongoCryptError: Unexpected field: 'expiration' originates from the libmongocrypt library when it has received an object that has different keys than it expects. It then re-throws the error wrapped as a NodeJS MongoCryptError, making the source of this error hard to find.

The word "expiration" eventually led me to the MONGODB_AWS specification and this resolved MongoDB JIRA ticket that explains how the client-side field-level encryption (CSFLE) implementation wasn't aligned with the rest of the codebase in terms of authentication processes.

In my case, libmongocrypt was throwing an authentication error. It wasn't able to find the AWS credentials needed to authenticate with the AWS KMS service in the EB env vars: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Why this was confusing for me

According the Mongo docs here, in the "Important!" box, the wording is:

To use an IAM role instead of an IAM user to authenticate your application

To me, this meant I could remove the credentials in the env vars because I assumed that there was a mechanism in place to authenticate somehow using the IAM Role I had set up and attached to my EC2 instance with appropriate access to my KMS resource.

But this isn't the case: you need to have both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY defined in your environment to avoid this error.