Why does Pycryptodome MAC check fail when encrypting and decrypting JSON files?

Question

I am trying to do encrypt some JSON data with AES-256, using a password hashed with pbkdf2_sha256 as the key. I want to store the data in a file, be able to load it up, decrypt it, alter it, encrypt it, store it, and repeat.

I am using the passlib and pycryptodome libraries with python 3.8. The following test occurs inside a docker container and throws an error I haven't been able to correct

Does anyone have any clues on how I can improve my code (and knowledge)?

Test.py:

import os, json
from Crypto.PublicKey import RSA
from Crypto.Cipher import AES
from passlib.hash import pbkdf2_sha256

def setJsonData(jsonData, jsonFileName):
    with open(jsonFileName, 'wb') as jsonFile:
        password = 'd'
        key = pbkdf2_sha256.hash(password)[-16:]

        data = json.dumps(jsonData).encode("utf8")
        cipher = AES.new(key.encode("utf8"), AES.MODE_EAX)
        ciphertext, tag = cipher.encrypt_and_digest(data)

        [ jsonFile.write(x) for x in (cipher.nonce, tag, ciphertext) ]

def getJsonData(jsonFileName):
   with open(jsonFileName, 'rb') as jsonFile:
        password = 'd'
        key = pbkdf2_sha256.hash(password)[-16:]

        nonce, tag, ciphertext = [ jsonFile.read(x) for x in (16, 16, -1) ]
        cipher = AES.new(key.encode("utf8"), AES.MODE_EAX, nonce)
        data = cipher.decrypt_and_verify(ciphertext, tag)

        return json.loads(data)


dictTest = {}
dictTest['test'] = 1

print(str(dictTest))
setJsonData(dictTest, "test")

dictTest = getJsonData("test")
print(str(dictTest))

Output:

{'test': 1}
Traceback (most recent call last):
  File "test.py", line 37, in <module>
    dictTest = getJsonData("test")
  File "test.py", line 24, in getJsonData
    data = cipher.decrypt_and_verify(ciphertext, tag)
  File "/usr/local/lib/python3.8/site-packages/Crypto/Cipher/_mode_eax.py", line 368, in decrypt_and_verify
    self.verify(received_mac_tag)
  File "/usr/local/lib/python3.8/site-packages/Crypto/Cipher/_mode_eax.py", line 309, in verify
    raise ValueError("MAC check failed")
ValueError: MAC check failed

Research:

• Looked into this answer, but I believe my verify() call is in the right place

• I noted that in the python docs, it says:

  loads(dumps(x)) != x if x has non-string keys.

  but, when I re-run the test with dictTest['test'] = 'a' I have the same error.

• I suspected the problem was the json formatting, so I did the same test with a string and didn't make the json.loads and json.dumps calls, but I have the same error

Answer 1

The problem here is that key = pbkdf2_sha256.hash(password)[-16:] hashes the key with a new salt each call. Therefore, the cipher used to encrypt and decrypt the cipher text is going to be different, yielding different data, and thus failing the integrity check.

I changed my key derivation function to the following:

h = SHA3_256.new()
h.update(password.encode("utf-8"))
key = h.digest()