Why does Python cryptography and xmlsec1 produce different signatures

Question

I am working on an integration with an xml based API. I am able to sign xml requests successfully using xmlsec1 like this...

xmlsec1 --sign --lax-key-search --privkey-pem test_privkey.pem,test_cert.pem --output xml/signed.xml xml/template.xml

template.xml: <root><Signature xmlns="http://www.w3.org/2000/09/xmldsig#"><SignedInfo><CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"></CanonicalizationMethod><SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"></SignatureMethod><Reference URI=""><Transforms><Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"></Transform><Transform Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"></Transform></Transforms><DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"></DigestMethod><DigestValue></DigestValue></Reference></SignedInfo><SignatureValue/></Signature></root>

(I've removed the actual content in an attempt to simplify/debug the issue explained below)

This works fine, but now I'd like to do this in Python...

# Load the private key
with open(KEY_PATH, "rb") as key_file:
    private_key = load_pem_private_key(key_file.read(), None)

# Load the XML file
tree = etree.parse(f"xml/{XML_FILENAME}")
root = tree.getroot()

# Create a Signature element
signature_element = etree.Element("Signature")
signature_element.attrib["xmlns"] = "http://www.w3.org/2000/09/xmldsig#"

# Create a SignedInfo element
signed_info_element = etree.SubElement(signature_element, "SignedInfo")

# Create a CanonicalizationMethod element
canonicalization_method_element = etree.SubElement(signed_info_element, "CanonicalizationMethod")
canonicalization_method_element.attrib["Algorithm"] = "http://www.w3.org/TR/2001/REC-xml-c14n-20010315"

# Create a SignatureMethod element
signature_method_element = etree.SubElement(signed_info_element, "SignatureMethod")
signature_method_element.attrib["Algorithm"] = "http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"

# Create a Reference element
reference_element = etree.SubElement(signed_info_element, "Reference")
reference_element.attrib["URI"] = ""

# Create Transforms
transforms_element = etree.SubElement(reference_element, "Transforms")
transform_element1 = etree.SubElement(transforms_element, "Transform")
transform_element1.attrib["Algorithm"] = "http://www.w3.org/2000/09/xmldsig#enveloped-signature"
transform_element2 = etree.SubElement(transforms_element, "Transform")
transform_element2.attrib["Algorithm"] = "http://www.w3.org/TR/2001/REC-xml-c14n-20010315"

# Create DigestMethod
digest_method_element = etree.SubElement(reference_element, "DigestMethod")
digest_method_element.attrib["Algorithm"] = "http://www.w3.org/2001/04/xmlenc#sha256"

# Compute the digest value for the entire XML document and add it to the DigestValue element
digest = hashlib.sha256(etree.tostring(root, method="c14n")).digest()
digest_value_element = etree.SubElement(reference_element, "DigestValue")
digest_value_element.text = b64encode(digest).decode()

# Canonicalize the SignedInfo XML
c14n_signed_info = etree.tostring(signed_info_element, method="c14n")

# Create a SHA256 digest of the SignedInfo
digest = hashlib.sha256(c14n_signed_info).digest()

# Sign the digest
signature = private_key.sign(
    digest,
    padding.PKCS1v15(),
    hashes.SHA256()
)

# Embed the SignatureValue in the Signature
signature_value_element = etree.SubElement(signature_element, "SignatureValue")
signature_value_element.text = b64encode(signature).decode()

# Add the Signature element to the root of the document
root.append(signature_element)

# Save the signed XML
tree = etree.ElementTree(root)
with open('xml/signed.xml', 'wb') as f:
    tree.write(f)

XML_FILENAME (simplified for debugging): <root></root>

When I submit the contents of signed.xml to the API it fails and this API provider is not able to help. In troubleshooting, I noticed that the SignatureValue is different between the Python and xmlsec1 versions (the digest value is the same).

Any ideas on why the SignatureValue would differ, or any other differences you see that would explain why it works with xmlsec1 and not with the Python code. I also tried the signxml Python library with the same results.

Thanks.

Answer 1

I have not looked closely but you appear to be adding elements to the XML before you do the digest. The digest should be across the canonical version of the original XML. The Signed info and signature is added after the digest is calculated.

Update:

Take a look at https://github.com/perl-net-saml2/perl-XML-Sig/blob/8cd5c375e1f29469c13b9925b28ab02cf1024468/lib/XML/Sig.pm#L319 if you can read perl :-). I have not run into any issues with signatures and I test against xmlsec1 regularly