I am reading a large zip file(500MB) from a URL using request.get(url).File contains one CSV file which is pretty huge in size. I am reading the response and writing the filestream into a zip file using fs.createWriteStream(zipFile). On close event of the fs.createWriteStream I have tried using adm-zip file with which i got "error invalid or unsupported zip format. no end header found" and with Unzipper npm package I am getting "invalid signature unzip ". Below is the code
const request = require('superagent');
const fs = require('fs');
const unzip = require('unzipper');
request.get(url).on('error', function(err) { console.log(err) }
.pipe(fs.createWriteStream(zipFile))
.on('close', function() {
const admZip = require('adm-zip');
console.log('start unzip');
var zip = new admZip(zipFile);
console.log('unzipping ' + uploadDir + "to");
zip.extractAllTo(uploadDir, true);
console.log('finished unzip');
with Unzipper
request.get(url)
.on('error', function(err) { console.log(err) }
.pipe(fs.createWriteStream(zipFile))
.on('close', function() {
fs.createReadStream(zipFile)
.pipe(unzipper.Extract({ path: UploadDir }));`enter code here`
})
Error is resolved. First step is to pipe the incoming readable response stream.
Once the read is done pipe will trigger the write stream. Then I am triggering the unzipping process on close event of the writestream.