Additional Octet String in Bouncycastle signed data message

Question

I am creating a CSCA Master list signed data but after successful creation of it, I see two Octet String instead of one in signed message. Please check the source code and output files

CMS using BC

// load master list signer key  
        PKCS8EncodedKeySpec spec = new PKCS8EncodedKeySpec(FileTools.readFiletoBuffer("CSCAFiles/MLS.key"));
        ASN1InputStream bIn = new ASN1InputStream(new ByteArrayInputStream(spec.getEncoded()));
        PrivateKeyInfo pki = PrivateKeyInfo.getInstance(bIn.readObject());
        String algOid = pki.getPrivateKeyAlgorithm().getAlgorithm().getId();

        PrivateKey MLSkey = KeyFactory.getInstance(algOid).generatePrivate(spec);

              // load master list signing cert
        X509Certificate MLSCert = CertTools.getCertfromByteArray(FileTools.readFiletoBuffer("CSCAFiles/MLSCert.der"),
                X509Certificate.class);

        // create master list
        X509CertificateHolder x509CertificateHolder = new X509CertificateHolder(MLSCert.getEncoded());

        String pubkeyAlgorithm = MLSCert.getPublicKey().getAlgorithm();
        String certAlgorithm = "";

        if (pubkeyAlgorithm.equals("RSA")) {
            certAlgorithm = "SHA256WITHRSA";
        } else if (pubkeyAlgorithm.equals("ECDSA")) {
            certAlgorithm = "SHA256WITHECDSA";
        } else if (pubkeyAlgorithm.equals("DSA")) {
            certAlgorithm = "SHA256WITHDSA";
        } else {
            certAlgorithm = "SHA256WITHRSA";
        }

        // csca cert
        X509Certificate cscaCert = CertTools.getCertfromByteArray(FileTools.readFiletoBuffer("CSCAFiles/CSCACert.der"),
                X509Certificate.class);

        // create master list content
        JcaX509CertificateHolder[] hollist = CertTools
                .convertToX509CertificateHolder(new X509Certificate[] { cscaCert });

        org.bouncycastle.asn1.x509.Certificate certList[] = new org.bouncycastle.asn1.x509.Certificate[1];
        certList[0] = hollist[0].toASN1Structure();
        //certList[1] = hollist[0].toASN1Structure();

        CscaMasterList ml = new CscaMasterList(certList);

        System.out.println(">>>>> encoded data : " + new java.math.BigInteger(1, ml.getEncoded()).toString(16));

        CMSTypedData message = new CMSProcessableByteArray(ICAOObjectIdentifiers.id_icao_cscaMasterList,
                ml.toASN1Primitive().getEncoded());   

        AlgorithmIdentifier sigAlgId = new DefaultSignatureAlgorithmIdentifierFinder().find(certAlgorithm);

        AlgorithmIdentifier digAlgId = new DefaultDigestAlgorithmIdentifierFinder().find(sigAlgId);
        AsymmetricKeyParameter privateKeyParameter = PrivateKeyFactory.createKey(MLSkey.getEncoded());

        BcContentSignerBuilder signBuilder = null;
        if (pubkeyAlgorithm.equals("RSA")) {
            signBuilder = new BcRSAContentSignerBuilder(sigAlgId, digAlgId);
        } else if (pubkeyAlgorithm.equals("EC")) {
            signBuilder = new BcECContentSignerBuilder(sigAlgId, digAlgId);
        } else if (pubkeyAlgorithm.equals("DSA")) {
            signBuilder = new BcDSAContentSignerBuilder(sigAlgId, digAlgId);
        } else {
            signBuilder = new BcRSAContentSignerBuilder(sigAlgId, digAlgId);
        }

        ContentSigner signer = signBuilder.build(privateKeyParameter);

        SignerInfoGeneratorBuilder signerInfoGeneratorBuilder = new SignerInfoGeneratorBuilder(
                new BcDigestCalculatorProvider());
        SignerInfoGenerator infoGenerator = signerInfoGeneratorBuilder.build(signer, x509CertificateHolder);

        CMSSignedDataGenerator dataGenerator = new CMSSignedDataGenerator();
        dataGenerator.addSignerInfoGenerator(infoGenerator);

        dataGenerator.addCertificate(x509CertificateHolder);
        dataGenerator.addCertificate(new X509CertificateHolder(FileTools.readFiletoBuffer("CSCAFiles/CSCACert.der")));

        CMSSignedData signedData = dataGenerator.generate(message, true);

        System.out.println(">>>>> ML data : " + new java.math.BigInteger(1, signedData.getEncoded()).toString(16));     

>>>>> encoded data :

308202b2020100318202ab308202a730820210a003020102021024c2de802df5eb9a5ffc421546f39635300d06092a864886f70d01010b0500303f3111300f06035504030c084353434154657374310d300b060355040b0c0464656d6f310e300c060355040a0c05647265616d310b3009060355040613024b52301e170d3139313033313032343830305a170d3334313032373032353830305a303f3111300f06035504030c084353434154657374310d300b060355040b0c0464656d6f310e300c060355040a0c05647265616d310b3009060355040613024b5230819f300d06092a864886f70d010101050003818d0030818902818100b23a5a61e46f522c5d0e43325b9485065cabb6db0d88e32bb6ceb0d32ebb740485458e1e4e1cdfe317829e3d785f2de66d0f270120919f46d759a64e944c3579661cf850714af2fd2088f5dc91b1838571802f7d4d7b61219da296a8ae7f55362aa28836cea220635660b0b8c8f3cd596234bbe5ccb04873aa3e301c05f86eb70203010001a381a33081a0300f0603551d130101ff040530030101ff300e0603551d0f0101ff040403020106302b0603551d1004243022800f32303139313033313032353830305a810f32303234313033313032353830305a301d0603551d0e0416041456d7d0f4db824006e0dc864129ddd6991b36c467301f0603551d2304183016801456d7d0f4db824006e0dc864129ddd6991b36c46730100603551d200409300730050603290101300d06092a864886f70d01010b05000381810027ebd5fe0fd6e8df20e485d07f5b3420cb9719d2bad3dec7db5611cb690087d26f398b4849dd7d99a2abf3a2a71f7b108941a12df1a8a9c42a9c48a0750fded7a5ba38f692780cb6663febfb28d879bb53155113d773b8349e0c658c4ddc5c3d1ec7f8d35d4e72223efaacfdc53698e9157ac4590c708f91346c286d1b7e4fe4

>>>>> ML data :

308006092a864886f70d010702a0803080020103310f300d0609608648016503040201050030800606678108010102a0802480048202b6308202b2020100318202ab308202a730820210a003020102021024c2de802df5eb9a5ffc421546f39635300d06092a864886f70d01010b0500303f3111300f06035504030c084353434154657374310d300b060355040b0c0464656d6f310e300c060355040a0c05647265616d310b3009060355040613024b52301e170d3139313033313032343830305a170d3334313032373032353830305a303f3111300f06035504030c084353434154657374310d300b060355040b0c0464656d6f310e300c060355040a0c05647265616d310b3009060355040613024b5230819f300d06092a864886f70d010101050003818d0030818902818100b23a5a61e46f522c5d0e43325b9485065cabb6db0d88e32bb6ceb0d32ebb740485458e1e4e1cdfe317829e3d785f2de66d0f270120919f46d759a64e944c3579661cf850714af2fd2088f5dc91b1838571802f7d4d7b61219da296a8ae7f55362aa28836cea220635660b0b8c8f3cd596234bbe5ccb04873aa3e301c05f86eb70203010001a381a33081a0300f0603551d130101ff040530030101ff300e0603551d0f0101ff040403020106302b0603551d1004243022800f32303139313033313032353830305a810f32303234313033313032353830305a301d0603551d0e0416041456d7d0f4db824006e0dc864129ddd6991b36c467301f0603551d2304183016801456d7d0f4db824006e0dc864129ddd6991b36c46730100603551d200409300730050603290101300d06092a864886f70d01010b05000381810027ebd5fe0fd6e8df20e485d07f5b3420cb9719d2bad3dec7db5611cb690087d26f398b4849dd7d99a2abf3a2a71f7b108941a12df1a8a9c42a9c48a0750fded7a5ba38f692780cb6663febfb28d879bb53155113d773b8349e0c658c4ddc5c3d1ec7f8d35d4e72223efaacfdc53698e9157ac4590c708f91346c286d1b7e4fe4000000000000a0803082044c308203b5a00302010202142710e65ec4b56813da4e0e1e0223b7d485aa64d9300d06092a864886f70d01010b0500303f3111300f06035504030c084353434154657374310d300b060355040b0c0464656d6f310e300c060355040a0c05647265616d310b3009060355040613024b52301e170d3139313033313032343830325a170d3239313032383032343830325a304f3121301f06035504030c184d61737465724c6973745369676e65724365727454657374310d300b060355040b0c0464656d6f310e300c060355040a0c05647265616d310b3009060355040613024b5230820122300d06092a864886f70d01010105000382010f003082010a0282010100cb4ef691bb600dd6f514569ed79853d56f053257ddcaf11cd7323824499d011d5cc7652e50789977922c76d8c0d937426ce81e74b5b4f52a419481aab713114916859c23a53ac3937ac22a73c1f31b281f74c5c31574ab2f5f270d31667ee3fe9f69d231957dd33a4c97a20c19c0cabcfddc467786c6be42c6fef962c00f44d17e5d50c39443a14d9f44baf89974e5b5620381e4c2096008e7994f4eb1ee70689fce105a22dc1316d4fea5753673c68cc1fc9b6636b088408672bc57ccbc3f2cd74855939c1ab6bf7cb9b7a56f9ace27c92abf431381659ce9d99326f8fd485a24306b7514b5c5903bf417f8a460aec2893f7913c184638641804891836844cb0203010001a38201af308201ab301f0603551d2304183016801456d7d0f4db824006e0dc864129ddd6991b36c46730818e06082b06010505070101048181307f302006082b060105050730028614687474703a2f2f6578616d706c652e636f6d2f30302006082b060105050730028614687474703a2f2f6578616d706c652e636f6d2f31303906082b06010505073001862d687474703a2f2f63612d646566696e65642e6f6373702e736572766963652e6c6f6361746f722e75726c2e7377302c0603551d110425302381214d61737465724c6973745369676e657243657274546573744064656d6f2e636f6d30140603551d250101ff040a3008060667810801010330570603551d1f0450304e3025a023a021861f687474703a2f2f7777772e64656d6f2e6f72672f6261722f6261722e63726c3025a023a021861f687474703a2f2f7777772e64656d6f2e6f72672f666f6f2f666f6f2e63726c301d0603551d0e041604148d2713fb3995e04b0dda8a02a1a3d9141ba01914302b0603551d1004243022800f32303139313033313032343830325a810f32303230303133313032343830325a300e0603551d0f0101ff040403020780300d06092a864886f70d01010b050003818100004f1211d31b962c3f7060ca29af5895593aaaecd4a0ec4657fc7e4e54d8f427771c0ff58db6ee5d11fe74074f0838cfbc541fd4a793d10baa4bd0b8b5a4aae8eb6eb330dafcae2f88f9f94981ccc8983565724365ea8dc793bf80b131ffa2f29e9aa4943597fcce72b0ddf2228c942ec4be3a3490935846cdef8664bf0bc5c1308202a730820210a003020102021024c2de802df5eb9a5ffc421546f39635300d06092a864886f70d01010b0500303f3111300f06035504030c084353434154657374310d300b060355040b0c0464656d6f310e300c060355040a0c05647265616d310b3009060355040613024b52301e170d3139313033313032343830305a170d3334313032373032353830305a303f3111300f06035504030c084353434154657374310d300b060355040b0c0464656d6f310e300c060355040a0c05647265616d310b3009060355040613024b5230819f300d06092a864886f70d010101050003818d0030818902818100b23a5a61e46f522c5d0e43325b9485065cabb6db0d88e32bb6ceb0d32ebb740485458e1e4e1cdfe317829e3d785f2de66d0f270120919f46d759a64e944c3579661cf850714af2fd2088f5dc91b1838571802f7d4d7b61219da296a8ae7f55362aa28836cea220635660b0b8c8f3cd596234bbe5ccb04873aa3e301c05f86eb70203010001a381a33081a0300f0603551d130101ff040530030101ff300e0603551d0f0101ff040403020106302b0603551d1004243022800f32303139313033313032353830305a810f32303234313033313032353830305a301d0603551d0e0416041456d7d0f4db824006e0dc864129ddd6991b36c467301f0603551d2304183016801456d7d0f4db824006e0dc864129ddd6991b36c46730100603551d200409300730050603290101300d06092a864886f70d01010b05000381810027ebd5fe0fd6e8df20e485d07f5b3420cb9719d2bad3dec7db5611cb690087d26f398b4849dd7d99a2abf3a2a71f7b108941a12df1a8a9c42a9c48a0750fded7a5ba38f692780cb6663febfb28d879bb53155113d773b8349e0c658c4ddc5c3d1ec7f8d35d4e72223efaacfdc53698e9157ac4590c708f91346c286d1b7e4fe400003182021a308202160201013057303f3111300f06035504030c084353434154657374310d300b060355040b0c0464656d6f310e300c060355040a0c05647265616d310b3009060355040613024b5202142710e65ec4b56813da4e0e1e0223b7d485aa64d9300d06096086480165030402010500a08195301506092a864886f70d01090331080606678108010102301c06092a864886f70d010905310f170d3139313130313030323033365a302d06092a864886f70d0109343120301e300d06096086480165030402010500a10d06092a864886f70d01010b0500302f06092a864886f70d010904312204202784558fbdeca49e7f905b161924b1b8306df980ed191907f1469ab46b379953300d06092a864886f70d01010b0500048201005d1ffa5ce0cef0304899b192eb873a8d9d656b28ed7fc1a4dfdeea9cc9325a9a228b9a7fb0962358c29d98f6a2b679e40fe5b918f615ecf12624764439c5e5b81665ceaca0c718e6b63bb57b8e8ea0b0579dc0b32923c9add0191b0864ff20b3245175ab7231e10c39d7867d7df1e712b81ba4cc4e0f300ede5c66ec73af6a6a3caf70358844a71c49d7b355d0e584d622c46dbdfc9ecb9a7fdea03a1a3e891f78c57420d2260b317c896312c982885d8f9438eddcef0c9ae19a775f0a7725440a6d4f0ad67daf2c8095e3b6668169e79f2e23b34876c2c8daf622bf0d0ec31cbd9259929dce951050564a7dcf92ad2f30d7945c2500f89653d3cfa2f4f7cc1a000000000000

Wrong Master list (using BC)

Expected Master list (using sun.security.pkcs)

----PKCS#7 using sun----

    // load master list signer key
        /* Check to see if this is in an EncryptedPrivateKeyInfo structure. */
        PKCS8EncodedKeySpec spec = new PKCS8EncodedKeySpec(FileTools.readFiletoBuffer("CSCAFiles/MLS.key"));
        /*
         * Now it's in a PKCS#8 PrivateKeyInfo structure. Read its Algorithm OID
         * and use that to construct a KeyFactory.
         */
        ASN1InputStream bIn = new ASN1InputStream(new ByteArrayInputStream(spec.getEncoded()));
        PrivateKeyInfo pki = PrivateKeyInfo.getInstance(bIn.readObject());
        String algOid = pki.getPrivateKeyAlgorithm().getAlgorithm().getId();

        PrivateKey MLSkey = KeyFactory.getInstance(algOid).generatePrivate(spec);

        X509Certificate MLSCert = CertTools.getCertfromByteArray(FileTools.readFiletoBuffer("CSCAFiles/MLSCert.der"),
                X509Certificate.class);

        // create the output stream
        ByteArrayOutputStream bOut = new ByteArrayOutputStream();

        bOut.write(FileTools.readFiletoBuffer("CSCAFiles/CSCACert.der"));
        bOut.write(FileTools.readFiletoBuffer("CSCAFiles/DSCert.der"));
        bOut.write(FileTools.readFiletoBuffer("CSCAFiles/DLSCert.der"));

        bOut.close();

        // create ml
        // set up the generator
        ////////////////////////
        X509CertificateHolder x509CertificateHolder = new X509CertificateHolder(MLSCert.getEncoded());

        String pubkeyAlgorithm = MLSCert.getPublicKey().getAlgorithm();
        String certAlgorithm = "";

        if (pubkeyAlgorithm.equals("RSA")) {
            certAlgorithm = "SHA256WITHRSA";
        } else if (pubkeyAlgorithm.equals("ECDSA")) {
            certAlgorithm = "SHA256WITHECDSA";
        } else if (pubkeyAlgorithm.equals("DSA")) {
            certAlgorithm = "SHA256WITHDSA";
        } else {
            certAlgorithm = "SHA256WITHRSA";
        }

        // csca cert
        X509Certificate cscaCert = CertTools.getCertfromByteArray(FileTools.readFiletoBuffer("CSCAFiles/CSCACert.der"),
                X509Certificate.class);

        // create master list content
        JcaX509CertificateHolder[] hollist = CertTools
                .convertToX509CertificateHolder(new X509Certificate[] { cscaCert });

        org.bouncycastle.asn1.x509.Certificate certList[] = new org.bouncycastle.asn1.x509.Certificate[2];
        certList[0] = hollist[0].toASN1Structure();
        certList[1] = hollist[0].toASN1Structure();

        CscaMasterList ml = new CscaMasterList(certList);

        ///////////////////////

        // Data to sign
        byte[] dataToSign = ml.getEncoded();

        // compute signature:
        Signature signature = Signature.getInstance(certAlgorithm);
        signature.initSign(MLSkey);
        signature.update(dataToSign);
        byte[] signedData = signature.sign();

        // load X500Name
        sun.security.x509.X500Name xName = new sun.security.x509.X500Name(cscaCert.getSubjectDN().getName());
        // load serial number
        BigInteger serial = hollist[0].getSerialNumber();
        // laod digest algorithm
        AlgorithmId digestAlgorithmId = new AlgorithmId(AlgorithmId.SHA_oid);
        // load signing algorithm
        AlgorithmId signAlgorithmId = new AlgorithmId(AlgorithmId.RSAEncryption_oid);

        // Create SignerInfo:
        SignerInfo sInfo = new SignerInfo(xName, serial, digestAlgorithmId, signAlgorithmId, signedData);
        // Create ContentInfo:
        ContentInfo cInfo = new ContentInfo(new ObjectIdentifier("2.23.136.1.1.2"),
                new DerValue(DerValue.tag_OctetString, dataToSign));
        // Create PKCS7 Signed data
        PKCS7 p7 = new PKCS7(new AlgorithmId[] { digestAlgorithmId }, cInfo,
                new java.security.cert.X509Certificate[] { cscaCert }, new SignerInfo[] { sInfo });
        // Write PKCS7 to bYteArray
        ByteArrayOutputStream bOut1 = new DerOutputStream();
        p7.encodeSignedData(bOut1);

        System.out.println(">>>>> ML data : " + new java.math.BigInteger(1, bOut1.toByteArray()).toString(16));         

Answer 1

Correction! While there is a difference in the way PKCS7 and CMS are created, as per comment the difference here is that Bouncy used 'constructed' encoding (with indefinite length, although that is less important here), see wikipedia. A 'constructed' item in ASN.1 has one tag-length prefix for the item and additional tag-length prefix(es) for one or more element(s) that make up the value of the constructed item.

$ openssl asn1parse <58653210.cms -inform der -i
    0:d=0  hl=2 l=inf  cons: SEQUENCE
    2:d=1  hl=2 l=   9 prim:  OBJECT            :pkcs7-signedData
   13:d=1  hl=2 l=inf  cons:  cont [ 0 ]
   15:d=2  hl=2 l=inf  cons:   SEQUENCE
   17:d=3  hl=2 l=   1 prim:    INTEGER           :03
   20:d=3  hl=2 l=  15 cons:    SET
   22:d=4  hl=2 l=  13 cons:     SEQUENCE
   24:d=5  hl=2 l=   9 prim:      OBJECT            :sha256
   35:d=5  hl=2 l=   0 prim:      NULL
   37:d=3  hl=2 l=inf  cons:    SEQUENCE
   39:d=4  hl=2 l=   6 prim:     OBJECT            :2.23.136.1.1.2
   47:d=4  hl=2 l=inf  cons:     cont [ 0 ]
   49:d=5  hl=2 l=inf  cons:      OCTET STRING
   51:d=6  hl=4 l= 694 prim:       OCTET STRING      [HEX DUMP]:308202B202010031 
[snip rest of (correct) body]
  749:d=6  hl=2 l=   0 prim:       EOC
  751:d=5  hl=2 l=   0 prim:      EOC
  753:d=4  hl=2 l=   0 prim:     EOC
  755:d=3  hl=2 l=inf  cons:    cont [ 0 ]
  757:d=4  hl=4 l=1100 cons:     SEQUENCE 
[snip certificate]
 1861:d=4  hl=4 l= 679 cons:     SEQUENCE 
[snip certificate]
 2544:d=4  hl=2 l=   0 prim:     EOC
 2546:d=3  hl=4 l= 538 cons:    SET
 2550:d=4  hl=4 l= 534 cons:     SEQUENCE 
[snip SignerInfo]
 3088:d=3  hl=2 l=   0 prim:    EOC
 3090:d=2  hl=2 l=   0 prim:   EOC
 3092:d=1  hl=2 l=   0 prim:  EOC

If we look at the actual bytes at offset 49 (in hex), we find

24 80 -- tag for OCTET STRING _constructed_, length indefinite
-- a constructed item contains (consists of) one or more following elements
04 82 02 B6 -- tag for OCTET STRING _primitive_, length 02B6
-- this is the only element within the constructed item

and at 49+6+0x2b6=749 we find 00 00 which is 'end of contents', terminating the constructed item. Thus the outer, constructed OCTET STRING actually consists of the one element which is a primitive OCTET STRING.

At a guess, Bouncy uses constructed+indefinite for the content and encapContentInfo, and the outer SignedData containing them, to allow them to be larger than fits in memory -- what in crypto is often called 'online' (a weird term for it) or a bit more sensibly 'streaming'. But it also uses this for the certificates and signerInfos components, which never have this issue -- maybe just for convenience?

The sun classes presumably didn't do this, although you didn't post the data to confirm (or the code used which could be looked at).

This is actually BER for two reasons; DER does not allow either constructed OCTET STRING or indefinite length.

---

(original but inapplicable answer for reference)

This is a small and easily missed difference between PKCS7 (the RSALabs original) and CMS (its IETFized successor). It was actually introduced in RFC 2630 in 1999, but not clearly documented until RFC 3369 sec 5.2.1 in 2002. Note that version (the Integer at beginning of SignedData, which is the outer Context-0 EXPLICIT SEQUENCE) also differs; in the Bouncy-created CMS version it is 3, while in the sun (for which you don't show the source, but is apparently PKCS7) it is 1. PKCS7 only defined version up to 1, so any (structure with) version > 1 must be CMS.