My code needs to parse heaps of JSON files, and those files are hosted on GitHub and only available bundled as 1 single ZIP file. Because the ZIP file is only about 80 MB, I want to keep the entire unzipping operation in memory.
I'm able to load the ZIP file into memory as a Data? variable, but I'm not able to find a way to unzip a Data variable in memory and then assign the unzipped file/data to some other variables. I've tried using ZIP Foundation, but its Archive type's initializers take only file URLs. I didn't try Zip, but its documentation shows that it takes file URLs as well.
Here is my code:
import Cocoa
import Alamofire
class ViewController: NSViewController {
override func viewDidLoad() {
super.viewDidLoad()
let zipURL = URL(string: "https://github.com/KSP-CKAN/CKAN-meta/archive/master.zip")!
AF.request(zipURL).validate().responseData { response in
var zipData: Data? = response.data
// Here I want to unzip `zipData` after unwrapping it.
}
}
}
I also looked into passing a Data variable off as a file, but failed to find a way to do it.
UPDATE (2019-12-01 05:00)
According to this pull request thread on ZIPFoundation, the feature I'm looking for will be included in the next release. I tried to use the feature's contributor's fork, but somehow Swift Package Manager wouldn't allow it.
Before finding this, I tried using Python's zipfile library through Swift-Python interoperability provided by PythonKit, but it didn't work out, because Foundation's Data in Swift can not be cast into a PythonObject type.
Apple's Compression framework also looked promising, but it seems to have a soft limit of 1 MB on compressed files. The compressed file I need is about 80 MB, way larger than 1 MB.
So far, ZIPFoundation is my most hopeful solution.
UPDATE (2019-12-01 06:00)
On another try, I was able to install microtherion's fork through Swift Package Manager. The following code should work:
import Cocoa
import Alamofire
import ZIPFoundation
... // ignoring irrelevant parts of the code
let zipURL = URL(string: "https://github.com/KSP-CKAN/CKAN-meta/archive/master.zip")!
AF.request(zipURL).validate().responseData { response in
// a Data variable that holds the raw bytes
var zipData: Data? = response.data
// an Archive instance created with the Data variable
var zipArchive = Archive(data: zipData!, accessMode: .read)
// iterate over the entries in the Archive instance, and extract each entry into a Data variable
for entry in zipArchive! {
var unzippedData: Data
do {
_ = try zipArchive?.extract(entry) {unzippedData($0)}
} catch {
...
}
...
}
}