Swift - zip/unzip Data in memory without any file system interaction

3.8k Views Asked by At

My code needs to parse heaps of JSON files, and those files are hosted on GitHub and only available bundled as 1 single ZIP file. Because the ZIP file is only about 80 MB, I want to keep the entire unzipping operation in memory.

I'm able to load the ZIP file into memory as a Data? variable, but I'm not able to find a way to unzip a Data variable in memory and then assign the unzipped file/data to some other variables. I've tried using ZIP Foundation, but its Archive type's initializers take only file URLs. I didn't try Zip, but its documentation shows that it takes file URLs as well.

Here is my code:

import Cocoa
import Alamofire

class ViewController: NSViewController {

    override func viewDidLoad() {
        super.viewDidLoad()

        let zipURL = URL(string: "https://github.com/KSP-CKAN/CKAN-meta/archive/master.zip")!

        AF.request(zipURL).validate().responseData { response in
            var zipData: Data? = response.data
            //  Here I want to unzip `zipData` after unwrapping it.
        }
    }

}

I also looked into passing a Data variable off as a file, but failed to find a way to do it.


UPDATE (2019-12-01 05:00)

According to this pull request thread on ZIPFoundation, the feature I'm looking for will be included in the next release. I tried to use the feature's contributor's fork, but somehow Swift Package Manager wouldn't allow it.

Before finding this, I tried using Python's zipfile library through Swift-Python interoperability provided by PythonKit, but it didn't work out, because Foundation's Data in Swift can not be cast into a PythonObject type.

Apple's Compression framework also looked promising, but it seems to have a soft limit of 1 MB on compressed files. The compressed file I need is about 80 MB, way larger than 1 MB.

So far, ZIPFoundation is my most hopeful solution.


UPDATE (2019-12-01 06:00)

On another try, I was able to install microtherion's fork through Swift Package Manager. The following code should work:

import Cocoa
import Alamofire
import ZIPFoundation

... //  ignoring irrelevant parts of the code


    let zipURL = URL(string: "https://github.com/KSP-CKAN/CKAN-meta/archive/master.zip")!

    AF.request(zipURL).validate().responseData { response in

        //  a Data variable that holds the raw bytes
        var zipData: Data? = response.data

        //  an Archive instance created with the Data variable
        var zipArchive = Archive(data: zipData!, accessMode: .read)

        //  iterate over the entries in the Archive instance, and extract each entry into a Data variable
        for entry in zipArchive! {
            var unzippedData: Data
            do {
                _ = try zipArchive?.extract(entry) {unzippedData($0)}
            } catch {
                ...
            }
            ...
        }

    }

0

There are 0 best solutions below