Dispatch queues and asynchronous RNCryptor

Question

This is a follow-up to Asynchronously decrypt a large file with RNCryptor on iOS

I've managed to asynchronously decrypt a large, downloaded file (60Mb) with the method described in this post, corrected by Calman in his answer.

It basically goes like this:

int blockSize = 32 * 1024;
NSInputStream *cryptedStream = [NSInputStream inputStreamWithFileAtPath:...];
NSOutputStream *decryptedStream = [NSOutputStream output...];

[cryptedStream open];
[decryptedStream open];

RNDecryptor *decryptor = [[RNDecryptor alloc] initWithPassword:@"blah" handler:^(RNCryptor *cryptor, NSData *data) {
    NSLog("Decryptor recevied %d bytes", data.length);
    [decryptedStream write:data.bytes maxLength:data.length];
    if (cryptor.isFinished) {
        [decryptedStream close];
        // call my delegate that I'm finished with decrypting
    }
}];

while (cryptedStream.hasBytesAvailable) {
    uint8_t buf[blockSize];
    NSUInteger bytesRead = [cryptedStream read:buf maxLength:blockSize];
    NSData *data = [NSData dataWithBytes:buf length:bytesRead];

    [decryptor addData:data];
    NSLog("Sent %d bytes to decryptor", bytesRead);
}

[cryptedStream close];
[decryptor finish];

However, I'm still facing a problem: the whole data is loaded in memory before being decrypted. I can see a bunch of "Sent X bytes to decryptor", and after that, the same bunch of "Decryptor recevied X bytes" in the console, when I'd like to see "Sent, received, sent, receives, ...".

That's fine for small (2Mb) files, or with large (60Mb) files on simulator; but on a real iPad1 it crashes due to memory constraints, so obviously I can't keep this procedure for my production app.

I feel like I need to send the data to the decryptor by using dispatch_async instead of blindly sending it in the while loop, however I'm completely lost. I've tried:

• creating my own queue before the while, and using dispatch_async(myQueue, ^{ [decryptor addData:data]; });
• the same, but dispatching the whole code inside of the while loop
• the same, but dispatching the whole while loop
• using RNCryptor-provided responseQueue instead of my own queue

Nothing works amongst these 4 variants.

I don't have a complete understanding of dispatch queues yet; I feel the problem lies here. I'd be glad if somebody could shed some light on this.

Answer 1

If you only want to process one block at a time, then only process a block when the first block calls you back. You don't need a semaphore to do that, you just need to perform the next read inside the callback. You might want an @autoreleasepool block inside of readStreamBlock, but I don't think you need it.

When I have some time, I'll probably wrap this directly into RNCryptor. I opened Issue#47 for it. I am open to pull requests.

// Make sure that this number is larger than the header + 1 block.
// 33+16 bytes = 49 bytes. So it shouldn't be a problem.
int blockSize = 32 * 1024;

NSInputStream *cryptedStream = [NSInputStream inputStreamWithFileAtPath:@"C++ Spec.pdf"];
NSOutputStream *decryptedStream = [NSOutputStream outputStreamToFileAtPath:@"/tmp/C++.crypt" append:NO];

[cryptedStream open];
[decryptedStream open];

// We don't need to keep making new NSData objects. We can just use one repeatedly.
__block NSMutableData *data = [NSMutableData dataWithLength:blockSize];
__block RNEncryptor *decryptor = nil;

dispatch_block_t readStreamBlock = ^{
  [data setLength:blockSize];
  NSInteger bytesRead = [cryptedStream read:[data mutableBytes] maxLength:blockSize];
  if (bytesRead < 0) {
    // Throw an error
  }
  else if (bytesRead == 0) {
    [decryptor finish];
  }
  else {
    [data setLength:bytesRead];
    [decryptor addData:data];
    NSLog(@"Sent %ld bytes to decryptor", (unsigned long)bytesRead);
  }
};

decryptor = [[RNEncryptor alloc] initWithSettings:kRNCryptorAES256Settings
                                         password:@"blah"
                                          handler:^(RNCryptor *cryptor, NSData *data) {
                                            NSLog(@"Decryptor recevied %ld bytes", (unsigned long)data.length);
                                            [decryptedStream write:data.bytes maxLength:data.length];
                                            if (cryptor.isFinished) {
                                              [decryptedStream close];
                                              // call my delegate that I'm finished with decrypting
                                            }
                                            else {
                                              // Might want to put this in a dispatch_async(), but I don't think you need it.
                                              readStreamBlock();
                                            }
                                          }];

// Read the first block to kick things off    
readStreamBlock();

Answer 2

Cyrille,

The reason your app is crashing due to memory constraints is that the RNCryptor buffer grows beyond the capabilities of the device.

Basically, you're reading the content of the file much faster than RNCryptor can handle it. Since it can't decrypt fast enough it buffers the incoming stream until it can process it.

I haven't yet have time to dive into the RNCryptor code and figure out exactly how it's using GCD to manage everything, but you can use a semaphore to force the reads to wait until the previous block was decrypted.

The code below can successfully decrypt a 225MB file on an iPad 1 without crashing.

It has a few issues that I'm not quite happy with, but it should give you a decent starting point.

Some things to note:

• I wrapped the internals of the while loop in an @autoreleasepool block to force the release of the data. Without it, the release won't happen until the while loop finishes. (Matt Galloway has a great post explaining it here: A Look under ARC's hood
• The call to dispatch_semaphore_wait blocks execution until a dispatch_semaphore_signal is received. This means no UI updates and the potential of the app freezing if you send one too many (thus the check for bytesRead > 0).

Personally I feel that there must be a better solution for this, but I haven't yet had the time to research it a bit more.

I hope this helps.

- (IBAction)decryptWithSemaphore:(id)sender {
    dispatch_semaphore_t semaphore = dispatch_semaphore_create(0);

    __block int total = 0;
    int blockSize = 32 * 1024;

    NSArray *docPaths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
    NSString *input = [[docPaths objectAtIndex:0] stringByAppendingPathComponent:@"zhuge.rncryptor"];
    NSString *output = [[docPaths objectAtIndex:0] stringByAppendingPathComponent:@"zhuge.decrypted.pdf"];

    NSInputStream *cryptedStream = [NSInputStream inputStreamWithFileAtPath:input];
    __block NSOutputStream *decryptedStream = [NSOutputStream outputStreamToFileAtPath:output append:NO];
    __block NSError *decryptionError = nil;

    [cryptedStream open];
    [decryptedStream open];

    RNDecryptor *decryptor = [[RNDecryptor alloc] initWithPassword:@"12345678901234567890123456789012" handler:^(RNCryptor *cryptor, NSData *data) {
        @autoreleasepool {
            NSLog(@"Decryptor recevied %d bytes", data.length);
            [decryptedStream write:data.bytes maxLength:data.length];
            dispatch_semaphore_signal(semaphore);

            data = nil;
            if (cryptor.isFinished) {
                [decryptedStream close];
                decryptionError = cryptor.error;
                // call my delegate that I'm finished with decrypting
            }
        }
    }];

    while (cryptedStream.hasBytesAvailable) {
        @autoreleasepool {
            uint8_t buf[blockSize];
            NSUInteger bytesRead = [cryptedStream read:buf maxLength:blockSize];
            if (bytesRead > 0) {
                NSData *data = [NSData dataWithBytes:buf length:bytesRead];

                total = total + bytesRead;
                [decryptor addData:data];
                NSLog(@"New bytes to decryptor: %d Total: %d", bytesRead, total);

                dispatch_semaphore_wait(semaphore, DISPATCH_TIME_FOREVER);
            }
        }
    }

    [cryptedStream close];
    [decryptor finish];

    dispatch_release(semaphore);

}

Answer 3

After spending the last 2 days trying to get my MBProgress hud to update its progress with Calman's code i came up with the following. The memory used still stays low and the UI updates

- (IBAction)decryptWithSemaphore:(id)sender {

__block int total = 0;
int blockSize = 32 * 1024;

NSArray *docPaths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
NSString *input = [[docPaths objectAtIndex:0] stringByAppendingPathComponent:@"zhuge.rncryptor"];
NSString *output = [[docPaths objectAtIndex:0] stringByAppendingPathComponent:@"zhuge.decrypted.pdf"];

NSInputStream *cryptedStream = [NSInputStream inputStreamWithFileAtPath:input];
__block NSOutputStream *decryptedStream = [NSOutputStream outputStreamToFileAtPath:output append:NO];
__block NSError *decryptionError = nil;
__block RNDecryptor *encryptor=nil;
NSDictionary *attributes = [[NSFileManager defaultManager] attributesOfItemAtPath:input error:NULL];
__block long long fileSize = [attributes fileSize];
[cryptedStream open];
[decryptedStream open];
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0);
dispatch_async(queue, ^{
    dispatch_semaphore_t semaphore = dispatch_semaphore_create(0);

encryptor = [[RNDecryptor alloc] initWithPassword:@"12345678901234567890123456789012" handler:^(RNCryptor *cryptor, NSData *data) {
    @autoreleasepool {
        NSLog(@"Decryptor recevied %d bytes", data.length);
        [decryptedStream write:data.bytes maxLength:data.length];
        dispatch_semaphore_signal(semaphore);

        data = nil;
        if (cryptor.isFinished) {
            [decryptedStream close];
            decryptionError = cryptor.error;
            [cryptedStream close];
            [encryptor finish];
            // call my delegate that I'm finished with decrypting
        }
    }
}];

while (cryptedStream.hasBytesAvailable) {
    @autoreleasepool {
        uint8_t buf[blockSize];
        NSUInteger bytesRead = [cryptedStream read:buf maxLength:blockSize];
        if (bytesRead > 0) {
            NSData *data = [NSData dataWithBytes:buf length:bytesRead];

            total = total + bytesRead;
            [encryptor addData:data];
            NSLog(@"New bytes to decryptor: %d Total: %d", bytesRead, total);

            dispatch_semaphore_wait(semaphore, DISPATCH_TIME_FOREVER);
            dispatch_async(dispatch_get_main_queue(), ^{
                HUD.progress = (float)total/fileSize;
            });

                           }
                           }
}
    });

}