Functions-framework (emulator) subscribing to Pub/Sub emulator

Question

I want to make a setup: My Dockerized App -> Pub/Sub Dockerized emulator <- Cloud functions Dockerized emulator

In steps:

1. My app publishes a message to Pub/Sub Docker emulator
2. Pub/Sub Docker emulator registers a message under topic
3. Cloud function/s subscribe to the topic and process

I want to have run every service locally and test the whole setup first in the development mode. I successfully set up the first two parts. I'm able to publish a message to Dockerized emulator and also get the data by subscribing to it.

After this, I wanted to add Cloud Functions (step 3) and let function/s subscribe to that topic. I Dockerized functions as well and put everything to docker-compose file so I don't need to think about network, environment, or setup steps.

My problem is that I don't know how to trigger the dockerized Cloud function when Pub/Sub emulator registers a new message.

Docker images I use:

1. My app is dockerized node.js Express server
2. I am using this image as my Pub/Sub emulator (it is simply Google's Pub/Sub emulator but dockerized). I can simply set topics and subscriptions via docker-compose file
3. I dockerized functions myself

Here is my docker-compose file:

version: '3.4'

services:
  myapp:
    build: ./myapp
    ports:
        - 8080:8080
    environment:
      NODE_ENV: development
      PUBSUB_EMULATOR_HOST: pubsub:8681
      PUBSUB_PROJECT_ID: pubsubproject
    volumes:
        - ./myapp/:/workspace
        - /workspace/node_modules
    depends_on: 
    - pubsub

  pubsub:
    image: messagebird/gcloud-pubsub-emulator
    environment:
      PUBSUB_PROJECT1: 'pubsubproject,published-events:published-events-pull'
    ports:
      - "8681:8681"

  functions:
    build:
      dockerfile: Dockerfile
      context: ./functions
    ports:
      - 8888:8888
    depends_on: 
      - pubsub
    environment:
      NODE_ENV: development
      PUBSUB_EMULATOR_HOST: pubsub:8681
      PUBSUB_PROJECT_ID: pubsubproject
    volumes:
      - ./functions/:/workspace
      - /workspace/node_modules

My app's code responsible for publishing (this works):

async publishToQueue(events: any[]) {
        const pubSub = new PubSub();
        const buffer = Buffer.from(JSON.stringify(events))

        try {
            const messageId = await pubSub.topic("published-events").publish(buffer);
            console.log(`Message ${messageId} published.`);
          } catch (error) {
            console.error(`Received error while publishing: ${error.message}`);
            process.exitCode = 1;
          }
    }

Cloud function's code (The code in this function is never triggered)

import { PubSub } from '@google-cloud/pubsub'

exports.helloPubSub = (data, context) => {   // <---- This function is never triggered
    console.log("FROM CLOUD FUNCTION")
    console.log(data)
    console.log(context)

    const pubSub = new PubSub()
    const subscription = pubSub.subscription("published-events-pull")

    const messageHandler = async message => {
        const data = JSON.parse(Buffer.from(message.data, 'base64').toString())
        console.log('Received data:')
        console.log(data)
        message.ack()
    }
    subscription.on('message', messageHandler)
};

package.json script (what is called from Dockerfile to start the function). dist/ folder is specified because I use TypeScript.

  "scripts": {
    "dev": "yarn run build:dev:init && yarn run build:dev | yarn run start:functions",
    "build:dev:init": "yarn run tsc ",
    "build:dev": "yarn run tsc -w ",
    "start:functions": "functions-framework --port=8888 --source=dist/ --target=helloPubSub --signature-type=event"
  }

Any help would be appreciated. I think people should have an option to test how multiple emulators communicate with each other and have a production-like feeling. I hope it is possible.

Answer 1

The function emulator code above needs to actually start the subscription to listen for messages from the Pub/Sub topic. Please note that Cloud Functions uses an underlying subscription to receive messages from Cloud Pub/Sub and trigger the user-defined function. Pub/Sub (and in this case the emulator) has no knowledge of any user-defined Cloud Functions.

To emulate this behavior, listen to messages from the Pub/Sub topic following this example and then invoke your desired user-defined function with constructed data and context parameters.