How can I read from S3 and then write to IPFS using Pinata?

Question

I have the following code which has the intention to retrieve a file from a S3 bucket and then directly pin it to IPFS using Pinata API:

const axios = require('axios');
const FormData = require('form-data')
const fs = require('fs')
const config = require('../config/config');
const { GetObjectCommand, S3Client } = require('@aws-sdk/client-s3')

const getObject = (region, bucket, key) => {

  const client = new S3Client({ region: region }) // Pass in opts to S3 if necessary

  return new Promise(async (resolve, reject) => {

    const getObjectCommand = new GetObjectCommand({
      Bucket: bucket,
      Key: key
    })

    try {
      const response = await client.send(getObjectCommand)

      // Store all of data chunks returned from the response data stream
      // into an array then use Array#join() to use the returned contents as a String
      let responseDataChunks = []

      // Handle an error while streaming the response body
      response.Body.once('error', err => reject(err))

      // Attach a 'data' listener to add the chunks of data to our array
      // Each chunk is a Buffer instance
      response.Body.on('data', chunk => responseDataChunks.push(chunk))

      // Once the stream has no more data, join the chunks into a string and return the string
      response.Body.once('end', () => resolve(responseDataChunks.join('')))

    } catch (err) {
      // Handle the error or throw
      return reject(err)
    }
  })
}

const pinFileToIPFS = async (region, bucket, fileName) => {
  const s3file = await getObject(region, bucket, fileName);
  const src = "/Users/matifalcone/GMG/NFT/Nissan/Cars/370Z NISMO 1/Metadata/1";
  const formData = new FormData();

  const file = fs.createReadStream(src)
  formData.append('file', file)

  const metadata = JSON.stringify({
    name: fileName,
  });
  formData.append('pinataMetadata', metadata);

  const options = JSON.stringify({
    cidVersion: 0,
  })
  formData.append('pinataOptions', options);

  try{
    const res = await axios.post(`${config.pinata.apiUrl}/pinning/pinFileToIPFS`, formData, {
      maxBodyLength: "Infinity",
      headers: {
        'Content-Type': `multipart/form-data; boundary=${formData._boundary}`,
        Authorization: `Bearer ${config.pinata.jwt}`
      }
    });
    console.log(res.data);
    return res.data
  } catch (error) {
    console.log(error);
  }
}

module.exports = {
  pinFileToIPFS
};

As you can see, the code is working right now but with a partial solution, where I get the file from my localhost. I am not yet being able to use s3file as input to formData.append yet because if I do, that's when Pinata fails with HTTP 400.

I suspect this is happening because formData.append expects a ReadStream and I am getting a string from the S3 client.

What is the right way to pass the contents of the S3 object to the Pinata API?

Answer 1

In some cases you might need to pass in a filename in the formdata in order for the API to accept it by appending it like so

formData.append('file', file, {
  filename: fileName
})

Tutorial blog post on streaming files from S3 to Pinata