decryption function for XChaCha20-Poly1305

Question

I am developing a service in Next.js that handles both file encryption and decryption using xchacha20-poly1305. While I have successfully implemented the encryption code, I am facing challenges with the decryption code. Could you please provide guidance on the most suitable decryption code for this encryption function? Additionally, I was also taking password as input from user

I am utilizing service workers to encrypt files in the browser, ensuring that it does not impact the main thread.

const [file, setFile] = useState();
const [password, setPassword] = useState();
navigator.serviceWorker.ready.then((reg) => {
    if (!reg || !reg.active) {
        setIsEncrypting(false);
        toast.error('Service worker is not ready or its not supported in your browser');
        return;
    }
    reg.active.postMessage({
        cmd: 'encryptFile',
            file,
            password
    });
});

I am using libsodium-wrappers-sumo library foe encryption

service-worker.js

self.addEventListener('install', (event) =>
    event.waitUntil(self.skipWaiting())
);

self.addEventListener('activate', (event) =>
    event.waitUntil(self.clients.claim())
);

const _sodium = require('libsodium-wrappers-sumo');
const STATIC_SIGNATURE = 'Encrypted By XXXXXXX';

(async () => {
    await _sodium.ready;
    const sodium = _sodium;

    addEventListener('message', async (e) => {
        switch (e.data.cmd) {
            case 'encryptFile':
                const startTime = performance.now();
                const { encryptedBlob, encryptedFileName } = await encryptFile(
                    e.data.file,
                    e.data.password
                );

                e.source.postMessage({
                    reply: 'encryptionFinished',
                    encryptedBlob,
                    encryptedFileName,
                });
                break;
        }
    });

    const encryptFile = async (file, password) => {
        // Generate encryption key
        const salt = sodium.randombytes_buf(sodium.crypto_pwhash_SALTBYTES);
        const key = sodium.crypto_pwhash(
            sodium.crypto_secretstream_xchacha20poly1305_KEYBYTES,
            sodium.from_string(password),
            salt,
            sodium.crypto_pwhash_OPSLIMIT_INTERACTIVE,
            sodium.crypto_pwhash_MEMLIMIT_INTERACTIVE,
            sodium.crypto_pwhash_ALG_ARGON2ID13
        );

        // Initialize encryption
        const { state, header } =
            sodium.crypto_secretstream_xchacha20poly1305_init_push(key);

        // Create a stream controller for chunked processing
        const streamController = new TransformStream();
        const writer = streamController.writable.getWriter();

        // Write signature, salt, and header to the stream
        const signature = sodium.from_string(STATIC_SIGNATURE);
        writer.write(signature);
        writer.write(salt);
        writer.write(header);

        // Encrypt file in chunks
        const chunkSize = 64 * 1024 * 1024;
        const reader = file.stream().getReader();

        while (true) {
            const { done, value } = await reader.read();

            if (done) {
                // Finalize encryption and close the stream
                const encryptedChunk =
                    sodium.crypto_secretstream_xchacha20poly1305_push(
                        state,
                        new Uint8Array(0),
                        null,
                        sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL
                    );
                writer.write(encryptedChunk);
                writer.close();

                // Get the encrypted file blob
                const encryptedBlob = await new Response(
                    streamController.readable
                ).blob();

                // send the encrypted file with the original filename + '.enc'
                const encryptedFileName = `${file.name}.enc`;
                return { encryptedBlob, encryptedFileName };
            }

            // Use chunkSize to control the size of each chunk
            for (let i = 0; i < value.length; i += chunkSize) {
                const chunk = value.slice(i, i + chunkSize);
                const encryptedChunk =
                    sodium.crypto_secretstream_xchacha20poly1305_push(
                        state,
                        new Uint8Array(chunk),
                        null,
                        sodium.crypto_secretstream_xchacha20poly1305_TAG_MESSAGE
                    );
                writer.write(encryptedChunk);
            }
        }
    };

})();

Now, I need a decryptFile function to first check the signature. It should verify if the file is encrypted by the same platform. After that, it should check whether the password provided by the user is correct to decrypt the file. Following that, the decoding process will occur, decrypting the file chunk by chunk. Finally, the function should return the decryptedBlob similar to how I implemented it in the encryptFile function, using a const chunkSize = 64 * 1024 * 1024; and also change the file name to .enc to non .enc

self.addEventListener('install', (event) =>
    event.waitUntil(self.skipWaiting())
);

self.addEventListener('activate', (event) =>
    event.waitUntil(self.clients.claim())
);

const _sodium = require('libsodium-wrappers-sumo');
const STATIC_SIGNATURE = 'Encrypted By XXXXXXX';

(async () => {
    await _sodium.ready;
    const sodium = _sodium;

    addEventListener('message', async (e) => {
        switch (e.data.cmd) {
            case 'encryptFile':
                ...
                break;
            case 'decryptFile':
                const {decryptedBlob, decryptedFileName} = await decryptFile(e.data.encFile, e.data.password);
                e.source.postMessage({
                    reply: 'decryptionFinished',
                    decryptedBlob,
                    decryptedFileName
                });
                break;
        }
    });

    const encryptFile = async (file, password) => {
        ...
    };

    const decryptFile = async (encFile, password) => {
        ... // help me to write this function
    };

})();

I am new to web cryptography and am currently reading the documentation for libsodium. However, I can't seem to find a solution. Please help me write the decryptFile function.

Also, if you could suggest any changes to the current code, that would be most welcome. Please provide recommendations for better performance and reliability.

const decryptFile = async (file, password) => {
        const signature = await file
            .slice(0, STATIC_SIGNATURE.length)
            .arrayBuffer();
        const decoder = new TextDecoder();

        if (decoder.decode(signature) !== STATIC_SIGNATURE) {
            throw new Error('Invalid signature');
        }

        const saltLength = sodium.crypto_pwhash_SALTBYTES;
        const saltBuffer = await file
            .slice(
                STATIC_SIGNATURE.length,
                STATIC_SIGNATURE.length + saltLength
            )
            .arrayBuffer();
        const salt = new Uint8Array(saltBuffer);

        const header = new Uint8Array(
            await file
                .slice(
                    STATIC_SIGNATURE.length + saltLength,
                    STATIC_SIGNATURE.length +
                        saltLength +
                        sodium.crypto_secretstream_xchacha20poly1305_HEADERBYTES
                )
                .arrayBuffer()
        );

        // Generate decryption key
        const key = sodium.crypto_pwhash(
            sodium.crypto_secretstream_xchacha20poly1305_KEYBYTES,
            sodium.from_string(password),
            salt,
            sodium.crypto_pwhash_OPSLIMIT_INTERACTIVE,
            sodium.crypto_pwhash_MEMLIMIT_INTERACTIVE,
            sodium.crypto_pwhash_ALG_ARGON2ID13
        );

        const { state_address, tag } =
            sodium.crypto_secretstream_xchacha20poly1305_init_pull(header, key);

        // Create a stream controller for chunked processing
        const streamController = new TransformStream();
        const writer = streamController.writable.getWriter();

        // Decrypt file in chunks
        const chunkSize = 64 * 1024 * 1024;
        const encryptedData = new Uint8Array(
            await file
                .slice(
                    STATIC_SIGNATURE.length +
                        saltLength +
                        sodium.crypto_secretstream_xchacha20poly1305_HEADERBYTES
                )
                .arrayBuffer()
        );
        let offset = 0;

        while (offset < encryptedData.length) {
            const chunk = new Uint8Array(
                encryptedData.slice(offset, offset + chunkSize)
            );

            const { message, tag: decryptedTag } =
                sodium.crypto_secretstream_xchacha20poly1305_pull(
                    state_address,
                    new Uint8Array(0),
                    chunk
                );

            if (
                decryptedTag ===
                sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL
            ) {
                break; // End of decryption
            }

            writer.write(message);
            offset += chunkSize;
        }

        writer.close();

        // Get the decrypted file blob
        const decryptedBlob = await new Response(
            streamController.readable
        ).blob();
        return decryptedBlob;
    };

This is what I have created so far, but it is giving an error and also not checking if the password is correct on a small chunk first.

The error look like this:

service-worker.js:20245 Uncaught (in promise) TypeError: state_address cannot be null or undefined

codesandbox

Answer 1

The current encryption method reads chunks using read() and splits them into smaller chunks of size chunkSize. Since the chunks read with read() are generally not multiples of chunkSize, the last chunk of a read() call is generally smaller than chunkSize, whereby the size of this chunk is unknown, as illustrated below:

read 1: r, r, r, r, s1, 
read 2: r, r, s2,
read 3: r, r, r, r, r, s3,
...
read n: r, r, r, sn

Here r are the chunks with size chunkSize and s1,...sn are shorter chunks of different sizes.

This corresponds to an effective chunk sequence of:

r, r, r, r, s1, r, r, s2, r, r, r, r, r, s3,..., r, r, r, sn

The intermediate smaller chunks make it impossible to correctly identify the ciphertext chunks due to their unknown length, so that decryption fails.

To avoid this, one approach is to suspend processing of the chunk that is too short, determine the next data using read(), append it to the chunk that is too short and then process the resulting data. This prevents chunks that are too short from occurring:

read 1: r, r, r, r,  
read 2: r, r, 
read 3: r, r, r, r, r, 
...
read n: r, r, r, sn

r, r, r, r, r, r, r, r, r, r, r,..., r, r, r, sn

So only the last chunk remains as a potentially shorter chunk, which is not a problem as this chunk is identified by the end of the data.

To implement this, the code of encryptFile() must be changed e.g. as follows:

async function encryptFile(file, password){

    // Define chunk size - must be agreed with the decrypting side
    const chunkSize = 192 * 1024;//64 * 1024 * 1024;

    // Generate encryption key
    const salt = sodium.randombytes_buf(sodium.crypto_pwhash_SALTBYTES);                    
    const key = sodium.crypto_pwhash(
        sodium.crypto_secretstream_xchacha20poly1305_KEYBYTES,
        sodium.from_string(password),
        salt,
        sodium.crypto_pwhash_OPSLIMIT_INTERACTIVE,
        sodium.crypto_pwhash_MEMLIMIT_INTERACTIVE,
        sodium.crypto_pwhash_ALG_ARGON2ID13
    );
    
    // Initialize encryption
    const { state, header } = sodium.crypto_secretstream_xchacha20poly1305_init_push(key);

    // Create a stream controller for chunked processing
    const streamController = new TransformStream();
    const writer = streamController.writable.getWriter();

    // Write signature, salt, and header to the stream
    const signature = sodium.from_string(STATIC_SIGNATURE);
    writer.write(signature);
    writer.write(salt);
    writer.write(header);                   

    // Encrypt file in chunks
    const reader = file.stream().getReader();
    let dataQueue = new Uint8Array(0) // Queue for data to be encrypted
    while (true) {
        const { done, value } = await reader.read();

        // Add the read data to the queue
        dataQueue = new Uint8Array([...dataQueue, ...new Uint8Array(value)]);
        
        if (done) {
        
            // Finalize encryption and close the stream
            let encryptedChunk = sodium.crypto_secretstream_xchacha20poly1305_push(
                state,
                new Uint8Array(dataQueue),
                null,
                sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL
            );
            writer.write(encryptedChunk);
            writer.close();

            // Get the encrypted file blob
            const encryptedBlob = await new Response(
                streamController.readable
            ).blob();

            // send the encrypted file with the original filename + '.enc'
            const encryptedFileName = `${file.name}.enc`;
            return { encryptedBlob, encryptedFileName };
        }

        // Use chunkSize to control the size of each chunk; if the last chunk is smaller than chunkSize 
        // (which is generally the case) it will not be encrypted; the last chunk then remains in the queue;
        // This prevents intermediate chunks that are smaller than chunkSize
        let dataQueueIsEmpty = true;
        for (let i = 0; i < dataQueue.length; i += chunkSize) {
            const chunk = dataQueue.slice(i, i + chunkSize);
            if (chunk.length == chunkSize) {
                const encryptedChunk = sodium.crypto_secretstream_xchacha20poly1305_push(
                    state,
                    new Uint8Array(chunk),
                    null,
                    sodium.crypto_secretstream_xchacha20poly1305_TAG_MESSAGE
                );
                writer.write(encryptedChunk);
            }
            else {
                dataQueue = chunk;
                dataQueueIsEmpty = false;
            }
        }
        if (dataQueueIsEmpty) {dataQueue = new Uint8Array(0)}
    }
}

---

Decryption must be adapted to the changed encryption. It must also be taken into account:

• A ciphertext chunk is larger than a plaintext chunk. The constant sodium.crypto_secretstream_xchacha20poly1305_ABYTES specifies the number of additional bytes. This must be considered during decryption when defining the chunk size: chunkSize (dec) = chunkSize(enc) + sodium.crypto_secretstream_xchacha20poly1305_ABYTES, see this example in the Libsodium documentation.
  This is not taken into account in the current code, which is another reason why the decryption fails.
• This example from the libsodium-wrapper-sumo documentation illustrates how to use sodium.crypto_secretstream_xchacha20poly1305_init_pull() and sodium.crypto_secretstream_xchacha20poly1305_pull(). Both are used incorrectly in the current code. This causes sodium.crypto_secretstream_xchacha20poly1305_init_pull() to return an undefined for state_address, which later leads to the posted error message when used in sodium.crypto_secretstream_xchacha20poly1305_pull().
• The encrypted data contains the signature, salt and header at the beginning. In the following implementation, a read() call (or several if necessary) is first made until this data has been determined. The remaining data is then processed before further data is determined using read() (this is not absolutely necessary, but this is how this implementation does it to avoid too large amounts of data).

The following code is a possible implementation:

async function decryptFile(file, password) {

    // Define chunk size - must be agreed with the enrypting side
    const chunkSize = 192 * 1024 + sodium.crypto_secretstream_xchacha20poly1305_ABYTES; //64 * 1024 * 1024;
                        
    // Get signature, salt and header
    const reader = file.stream().getReader();
    let dataQueue = new Uint8Array(0); // Queue for data to be encrypted
    while (dataQueue.byteLength < STATIC_SIGNATURE.length + sodium.crypto_pwhash_SALTBYTES + sodium.crypto_secretstream_xchacha20poly1305_HEADERBYTES) {
        const { done, value } = await reader.read();

        // Add the read data to the queue
        dataQueue = new Uint8Array([...dataQueue, ...new Uint8Array(value)]);
    }
    const signature = dataQueue.slice(0, STATIC_SIGNATURE.length);
    const salt = dataQueue.slice(STATIC_SIGNATURE.length, STATIC_SIGNATURE.length + sodium.crypto_pwhash_SALTBYTES);
    const header = dataQueue.slice(STATIC_SIGNATURE.length + sodium.crypto_pwhash_SALTBYTES, STATIC_SIGNATURE.length + sodium.crypto_pwhash_SALTBYTES + sodium.crypto_secretstream_xchacha20poly1305_HEADERBYTES);
    dataQueue = dataQueue.slice(STATIC_SIGNATURE.length + sodium.crypto_pwhash_SALTBYTES + sodium.crypto_secretstream_xchacha20poly1305_HEADERBYTES);
    
    // Generate decryption key
    const key = sodium.crypto_pwhash(
        sodium.crypto_secretstream_xchacha20poly1305_KEYBYTES,
        sodium.from_string(password),
        salt,
        sodium.crypto_pwhash_OPSLIMIT_INTERACTIVE,
        sodium.crypto_pwhash_MEMLIMIT_INTERACTIVE,
        sodium.crypto_pwhash_ALG_ARGON2ID13
    );
    
    // Initialize decryption
    let state = sodium.crypto_secretstream_xchacha20poly1305_init_pull(header, key);

    // Create a stream controller for chunked processing
    const streamController = new TransformStream();
    const writer = streamController.writable.getWriter();

    // Loop for large chunks that were fetched with read() (containing multiple small chunks with size chunkSize)
    let dataWithHeader = true;
    while (true) {
    
        let done; 
        if (!dataWithHeader){
            // Add read data to queue
            let data = await reader.read();
            done = data.done;
            dataQueue = new Uint8Array([...dataQueue, ...new Uint8Array(data.value)]);
        } else {
            // Skip adding as queue still filled
            dataWithHeader = false;
            done = false;
        }
        
        if (done) {                     
            // Finalize decryption and close the stream
            let decryptedChunk = sodium.crypto_secretstream_xchacha20poly1305_pull(
                state,
                new Uint8Array(dataQueue)
            );
            // optional check: decryptedChunk.tag must be sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL
            writer.write(decryptedChunk.message);
            writer.close();
            // Get the decrypted file blob
            const decryptedBlob = await new Response(
                streamController.readable
            ).blob();
            // Send the decrypted file with the original filename + '.enc'
            const decryptedFileName = `${file.name}.enc`;
            return { decryptedBlob, decryptedFileName };
        }

        // Loop for small chunks with size chunkSize: Split the large chunks in chunks of size chunkSize.  
        // If the last chunk is smaller than chunkSize (which is generally the case) it will not be decrypted
        // and remains in the queue. This prevents intermediate chunks that are smaller than chunkSize.
        let dataQueueIsEmpty = true;
        for (let i = 0; i < dataQueue.length; i += chunkSize) {
            const chunk = dataQueue.slice(i, i + chunkSize);
            if (chunk.length == chunkSize) {
                const decryptedChunk = sodium.crypto_secretstream_xchacha20poly1305_pull(
                    state,
                    new Uint8Array(chunk),
                );
                // optional check: decryptedChunk.tag must be sodium.crypto_secretstream_xchacha20poly1305_TAG_MESSAGE
                writer.write(decryptedChunk.message);
            }
            else {
                dataQueue = chunk;
                dataQueueIsEmpty = false;
            }
        }
        if (dataQueueIsEmpty) {dataQueue = new Uint8Array(0);}
    }
};

---

Test: I have successfully tested encryption and decryption for a file of 14072985 bytes and a chunk size of 192 * 1024 bytes. With these parameters, several read calls are performed whose read data does not correspond to multiples of chunkSize, so that this test also proves the correct processing of smaller chunks.

---

Edit:

Since the primary goal was to have a working logic and to fix the bugs, the code does not contain any error handling and is not performance-optimized!
I.e. the exception handling must therefore be added according to your requirements as well as a performance optimization.

---

Regarding the password: crypto_secretstream_xchacha20poly1305_pull() returns message and tag in case of success and false in case of an error (e.g. invalid/incomplete/corrupted ciphertext, wrong password etc.).
In your code, there are generally several crypto_secretstream_xchacha20poly1305_pull() calls. If one of these calls returns false, the data has been tampered with, intentionally or unintentionally, and the decryption must be considered failed (and all data should be discarded as a precaution).

In the case of a wrong password, already the first call will return false, so that the error is quickly recognized.

---

Regarding the performance issue: The main reason for the performance problem is dataQueue and the copy operations used to fill it, i.e. the line: dataQueue = new Uint8Array([...dataQueue, ...new Uint8Array(value)]).
These copy operations become increasingly inperformant as the data size increases. This can be easily verified by outputting the execution time for this line together with dataQueue.length:

This problem is basically related to the use of a fixed chunk size. The fixed chunk size requires (because of the shorter chunks) a reorganization of the chunks into chunks of the given chunk size, which is associated with copying operations, appending etc.
I have solved this problem in my sample implementation mainly for the sake of simplicity with dataQueue.

What can be done to improve performance? First of all, the existing code can be performance-optimized. E.g. before filling dataQueue, it could be checked whether it is empty. If this is the case, dataQueue = value is sufficient instead of the expensive copying process. There may be further optimization options of this kind.

Such optimizations should require the least effort, but may not be sufficient. It would then be necessary to consider how the small chunks and the new data read in with read() could be converted into chunks of the given chunk size as efficiently as possible, i.e. with as few copy operations and/or as little copied data as possible, so that dataQueue can be dispensed with.
For this, e.g. the first chunk after a new read() call could be filled with the data of the last, too short chunk (which must be saved for this purpose somewhere) and the rest with the data of the new read() call. Subsequent chunks are then only filled with the data from the new read() call. The filling would therefore be more direct here and would not run via dataQueue. Although copying processes are also necessary here, the amount of data should be smaller.
A further optimization would of course be if copying/appending itself were more performant. Possibly, e.g. the use of regular arrays instead of typed arrays or something similar would provide a performance gain.
All in all, however, the optimizations described in this section require a larger amount of code changes and trial and error.

Another option, but a completely different approach, would be to dispense with the fixed chunk size and store the chunk size information unencrypted in the ciphertext (similar to signature, salt and header), e.g. by storing the chunk size in the first 4 bytes before the relevant chunk. This would make it possible to identify the chunks during decryption. There would be no need to reorganize the chunks. As the order of the chunks is checked during decryption, there is no vulnerability associated with this approach. However, this solution practically amounts to a new implementation and therefore involves the greatest change effort.