Writing file in /tmp in a Firebase Function does not work

Question

I am writing a Firebase function that exposes an API endpoint using express. When the endpoint is called, it needs to download an image from an external API and use that image to make a second API call. The second API call needs the image to be passed as a readableStream. Specifically, I am calling the pinFileToIPFS endpoint of the Pinata API.

My Firebase function is using axios to download the image and fs to write the image to /tmp. Then I am using fs to read the image, convert it to a readableStream and send it to Pinata.

A stripped-down version of my code looks like this:

const functions = require("firebase-functions");
const express = require("express");
const axios = require("axios");
const fs = require('fs-extra')
require("dotenv").config();
const key = process.env.REACT_APP_PINATA_KEY;
const secret = process.env.REACT_APP_PINATA_SECRET;
const pinataSDK = require('@pinata/sdk');
const pinata = pinataSDK(key, secret);
const app = express();

const downloadFile = async (fileUrl, downloadFilePath) => {
  try {
      const response = await axios({
        method: 'GET',
        url: fileUrl,
        responseType: 'stream',
      });


    // pipe the result stream into a file on disc
    response.data.pipe(fs.createWriteStream(downloadFilePath, {flags:'w'}))

    // return a promise and resolve when download finishes
    return new Promise((resolve, reject) => {
      response.data.on('end', () => {
        resolve()
      })

      response.data.on('error', () => {
        reject()
      })
    })

  } catch (err) { 
    console.log('Failed to download image')
    console.log(err);
    throw new Error(err);
  }
}; 

app.post('/pinata/pinFileToIPFS', cors(), async (req, res) => {
  const id = req.query.id;

  var  url = '<URL of API endpoint to download the image>';

  await fs.ensureDir('/tmp');

  if (fs.existsSync('/tmp')) {
    console.log('Folder: /tmp exists!')
  } else {
    console.log('Folder: /tmp does not exist!')
  }

  var filename = '/tmp/image-'+id+'.png';
  downloadFile(url, filename);

  if (fs.existsSync(filename)) {
    console.log('File: ' + filename + ' exists!')
  } else {
    console.log('File: ' + filename + ' does not exist!')
  }

  var image = fs.createReadStream(filename);

  const options = {
    pinataOptions: {cidVersion: 1}
  };

  pinata.pinFileToIPFS(image, options).then((result) => {
      console.log(JSON.stringify(result));
      res.header("Access-Control-Allow-Origin", "*");
      res.header("Access-Control-Allow-Headers", "Authorization, Origin, X-Requested-With, Accept");
      res.status(200).json(JSON.stringify(result));   
      res.send();
  }).catch((err) => {
      console.log('Failed to pin file');
      console.log(err);
      res.status(500).json(JSON.stringify(err));   
      res.send();
  });

});

exports.api = functions.https.onRequest(app);

Interestingly, my debug messages tell me that the /tmp folder exists, but the file of my downloaded file does not exist in the file system. [Error: ENOENT: no such file or directory, open '/tmp/image-314502.png']. Note that the image can be accessed correctly when I manually access the URL of the image.

I've tried to download and save the file using many ways but none of them work. Also, based on what I've read, Firebase Functions allow to write and read temp files from /tmp.

Any advice will be appreciated. Note that I am very new to NodeJS and to Firebase, so please excuse my basic code.

Many thanks!

Answer 1

I was not able to see you are initializing the directory as suggested in this post:

const bucket = gcs.bucket(object.bucket);
const filePath = object.name;
const fileName = filePath.split('/').pop();
const thumbFileName = 'thumb_' + fileName;

const workingDir = join(tmpdir(), `${object.name.split('/')[0]}/`);//new
const tmpFilePath = join(workingDir, fileName);
const tmpThumbPath = join(workingDir, thumbFileName);

await fs.ensureDir(workingDir);

Also, please consider that if you are using two functions, the /tmp directory would not be shared as each one has its own. Here is an explanation from Doug Stevenson. In the same answer, there is a very well explained video about local and global scopes and how to use the tmp directory:

Cloud Functions only allows one function to run at a time in a particular server instance. Functions running in parallel run on different server instances, which have different /tmp spaces. Each function invocation runs in complete isolation from each other. You should always clean up files you write in /tmp so that they don't accumulate and cause a server instance to run out of memory over time.

I would suggest using Google Cloud Storage extended with Cloud Functions to achieve your goal.