I have a situation where I need to move some data from one table to another in bigquery with some processing in middle. I have created a pipeline with Node.js client library which runs query for each value, gets data and process it and store to other table. I have a list of around 8000 values to match. Tables are partitioned and for each value it processes around 6 GB data in individual run. So, should I use the job as in client library or run it once in console and download the results then process. So basically it is:
SELECT * FROM table_name WHERE DATE(timestamp) = '2023-07-20' AND id IN ('1','2','3',....,'8000');
v/s
SELECT * FROM table_name WHERE DATE(timestamp) = '2023-07-20' AND id = '1';
.
.
.
SELECT * FROM table_name WHERE DATE(timestamp) = '2023-07-20' AND id = '8000';
I am more concerned about the cost rather than performance.
I tried googling it but didn't find a good reason when to use what. For me, client library works fine and easy to use and will go with it if both situations ends in same pool when cost is concerned.