iOS Common Crypto Decrypt RC4 Encrypted video file Memory crash

Question

I am decrypting a video file which works perfectly for small sizes files but for files above 300mb, there is memory crash. The code is as below : I checked the start byte value it goes uptill 315mb and then crashes, my file is sized at 350mb.

It works for few iphones and crashes for few, The best solution was do it in chunks to avoid memory issue but it crashes doing that too.

#define kChunkSizeBytes (1024*1024) // 1 MB

@implementation NSMutableData (Crypto)
   -(BOOL) doCrypto:(NSString *)key operation: (CCOperation) operation
{

    //Keeping it 32 as per our key
    char keyPtr[512 + 1]; // room for terminator (unused)
    bzero(keyPtr, sizeof(keyPtr));     // fill with zeroes (for padding)

    // Fetch key data

    if (![key getCString:keyPtr maxLength:sizeof(keyPtr) encoding:NSUTF8StringEncoding]) {return FALSE;} // Length of 'key' is bigger than keyPtr

    CCCryptorRef cryptor;




    CCCryptorStatus cryptStatus = CCCryptorCreate(operation, kCCAlgorithmRC4, 0,
                                                  keyPtr, key.length,
                                                  NULL, // IV - needed?
                                                  &cryptor);

    if (cryptStatus != kCCSuccess) { // Handle error here
        return FALSE;
    }

    size_t dataOutMoved;
    size_t dataInLength = kChunkSizeBytes; // #define kChunkSizeBytes (16)
    size_t dataOutLength = CCCryptorGetOutputLength(cryptor, dataInLength, FALSE);
    size_t totalLength = 0; // Keeps track of the total length of the output buffer
    size_t filePtr = 0;   // Maintains the file pointer for the output buffer
    NSInteger startByte; // Maintains the file pointer for the input buffer

    char *dataIn = malloc(dataInLength);
    char *dataOut = malloc(dataOutLength);
    NSRange bytesRange = NSMakeRange((NSUInteger) 0, (NSUInteger) 0);

    for (startByte = 0; startByte <= [self length]; startByte += kChunkSizeBytes) {
            if ((startByte + kChunkSizeBytes) > [self length]) {
                dataInLength = [self length] - startByte;
            }
            else {
                dataInLength = kChunkSizeBytes;
            }

            // Get the chunk to be ciphered from the input buffer
            bytesRange = NSMakeRange((NSUInteger) startByte, (NSUInteger) dataInLength);
            [self getBytes:dataIn range:bytesRange];
            cryptStatus = CCCryptorUpdate(cryptor, dataIn, dataInLength, dataOut, dataOutLength, &dataOutMoved);

            if (startByte >= 203728200) {
                NSLog(@"%ld",(long)startByte);
            }
            if (dataOutMoved != dataOutLength) {
                NSLog(@"dataOutMoved (%d) != dataOutLength (%d)", dataOutMoved, dataOutLength);
            }

            if ( cryptStatus != kCCSuccess)
            {
                NSLog(@"Failed CCCryptorUpdate: %d", cryptStatus);
            }

            // Write the ciphered buffer into the output buffer
            bytesRange = NSMakeRange(filePtr, (NSUInteger) dataOutMoved);
            [self replaceBytesInRange:bytesRange withBytes:dataOut];
            totalLength += dataOutMoved;

            filePtr += dataOutMoved;

    }

    // Finalize encryption/decryption.
    cryptStatus = CCCryptorFinal(cryptor, dataOut, dataOutLength, &dataOutMoved);
    totalLength += dataOutMoved;

    if ( cryptStatus != kCCSuccess)
    {
        NSLog(@"Failed CCCryptorFinal: %d", cryptStatus);
    }

    // In the case of encryption, expand the buffer if it required some padding (an encrypted buffer will always be a multiple of 16).
    // In the case of decryption, truncate our buffer in case the encrypted buffer contained some padding
    [self setLength:totalLength];

    // Finalize the buffer with data from the CCCryptorFinal call
    NSRange bytesNewRange = NSMakeRange(filePtr, (NSUInteger) dataOutMoved);
    [self replaceBytesInRange:bytesNewRange withBytes:dataOut];

    CCCryptorRelease(cryptor);

    free(dataIn);
    free(dataOut);

    return 1;
}
@end

Answer 1

If replaceBytesInRange:bytesRange is causing the crash, then my first suggestion for how to avoid the crash is to add error checking for the preceding function calls that it depends upon.

For example, in those cases where it crashes, maybe bytesRange is not a valid / useable value. Maybe its dataOut that is not valid / usable. Your code in your question sets those values from function calls, but does not check the return value for error conditions / error indicators / invalid values.

It might be a related dependent function call. e.g., cryptStatus is set with a call to CCCryptorUpdate(), which has dataOut as an input parameter. I don't know Objective-C, and I'm not familiar with the function CCCryptorUpdate(), but it looks like it would affect / populate dataOut. If it's actually returning an error, then dataOut is probably not in a usable state by the time you use it on the replaceBytesInRange line. Checking the return value of cryptStatus might flag conditions when you should not proceed to use dataOut in later calls.

Which brings me to another thing I noticed : you do check a couple things, but only log them. The checks for if (dataOutMoved != dataOutLength) and for (cryptStatus != kCCSuccess) look like things that should stop execution, or break out of the loop, or something like that, not just log the occurrence.

Another thing I see is that dataOut is malloc()'d once, not cleared, and used repeatedly. This can be entirely valid in some circumstances, but it can also cause exactly the kind of error you are seeing. Does anything in your code assume anything about the content of dataOut? I'm thinking along the lines of C string operations that assume null-termination. If one is not careful, null terminators that are assumed to be there (or maybe are in fact there at first) can be overwritten if, say, the entire buffer is filled. Again, I don't know Objective-C and these functions so well, so this is not meant as a specific statement, but more of an analogy of the kind of thing that might be happening.

TL; DR : Add error checking to each of your function calls, and respond accordingly (break, exit, retry, whatever) so that no later function call attempts to use values that indicate errors or are otherwise invalid.

I'll wager that with added error checking you will (1) stop the crashes, and (2) learn something specific about why it is that files larger than a certain amount yield these crashes.