Require bigquery in Bigquery DataForm script file

Question

Context

I am using the Dataform-Preview in BigQuery. As it is a node.js project I wanted to work around the constraints Dataform poses regarding retrieving/loading data from BQ-tables to use it for advanced processing.

Thus, I npm install --save @google-cloud/bigquery adding it to the package.json and also installed it in the web-IDE.

Error

I created a sample.js file inside the include-directory and required the package as usual.

// Content of sample.js
const {BigQuery} = require('@google-cloud/bigquery');

Normally, there should be no issue with requiring packages. Anyhow, requiring the bigquery itself fails with this error:

Error in includes/sample.js Failed to resolve stream

Investigating a little further shows that it is not a BQ-specific error. Some packages natively available in Node.js (e.g. fs or child_process) fail to import while packages like date-and-time installed using npm are requirable.

Thus, I assume that, like with other Google environments, BQs Dataform might have some environment pre-created messing with installations; anyhow, this is just an assumption. Also, according to the BQ Dataform docs it actually should work.

Question

How can I successfully import packages installed via npm? Also, I would be very interesting if anybody knew why the described steps fail even though they are according to the docs.

Edit 27.03.23

As side-note: Outside the Web-IDE the Node.js-project works fine and package-requiring does not pose any issue; thus, the issue should not be about a broken project but rather related to the Dataform instance in BigQuery.

Answer 1

When you create your first Dataform repository, Dataform automatically generates a service account. Dataform uses the service account to interact with BigQuery on your behalf.

Your Dataform service account ID is in the following format:

service-YOUR_PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com

Also grant the following permission to the above-mentioned service account:

• BigQuery Job User role.
• BigQuery Data Editor role.
• BigQuery Data Viewer role.

And talking about hierarchy take the .js file outside the include folder.

I hope your issue is resolved by this

Answer 2

It seems that you cannot install emitter because the installation becomes too large and it times out. According to google documentation there is a limit of 200MB. It would be nice to find a way to work around this problem in order to query tables within javascript in dataforma js files.