Google Cloud Dataflow error NoSuchMethodException: No such function

Question

I am using a dataflow function to transform pubsub messages in the form of json written as a string to submit into a bigquery table with the correct schemas in place.

I use the following UDF function which I have tested to work correctly in the google cloud shell.

function networkprocess(inJson) {
  data = JSON.parse(inJson);

  data ["Network_Details"] = JSON.parse(data["Network_Details"]);
  index = data["Network_Details"]["IpAddress"].indexOf(data["source_internal_ip"]);
  if (index == -1) {
      index = 0;
    }
  for (var key in data["Network_Details"]){
      data[key] = (data["Network_Details"][key][index]);
    }
  

  delete  data ["Network_Details"];
 return JSON.stringify(data);
}

However this dataflow job keeps outputting failed records to a failed record table with the following error:

java.lang.NoSuchMethodException: No such function networkprocess at org.openjdk.nashorn.api.scripting.ScriptObjectMirror.callMember(ScriptObjectMirror.java:192) at .......

............ a lot of "at" pointers....

I have tried creating new UDFs from scracth as well as using the standard function google provides when clicking create UDF during the job set up (in GUI) which is just:

function process(inJson) {

 return JSON.stringify(data);
}

and it they all return the same error that it cannot find the function.

I am deploying this via GUI or terraform but its the same result.

Answer 1

It seems like the function is not loaded properly, somehow.

You saved the JavaScript file in a GCS bucket, and pointed to it using the right path?

Does the service account that is being used in Dataflow has the right set of permissions to fetch that file?

Answer 2

Might sound like a silly suggestion, but maybe it is some error on Google's side and you should just contact Google Support directly.

Answer 3

I had the same problem. Check if you file name ends with .js. In my case that was the problem. I had this gs://bucket-name/file-name and I had the same error than you. I changed to gs://bucket-name/file-name.js and the problem was solved.