How to select large result set from Google BigQuery using .net api client library without pagination

Question

I am trying to select the large result set (~200K) from Google BigQuery using .net client library.

I able to get only ~4K rows at single shot (request & response), Google BigQuery offers PageToken to get next set of results. I am able to select all the rows from big query through multiple request to Google Bigquery (~50 Request & Response).

My Question: If there is possibility to select large result set in single response? if yes kindly share your code (even JAVA code also is fine).

Answer 1

There is no way to run a query and select a large response in a single shot. You can either paginate the results, or if you can create a job to export to files, then use the files generated in your app. Exporting is free.

Step to run a large query and export results to files stored on GCS:

1) Set allowLargeResults to true in your job configuration. You must also specify a destination table with the allowLargeResults flag.

Example:

"configuration": 
  {
    "query": 
    {
      "allowLargeResults": true,
      "query": "select uid from [project:dataset.table]"
      "destinationTable": [project:dataset.table]

    }
  }

2) Now your data is in a destination table you set. You need to create a new job, and set the export property to be able to export the table to file(s). Exporting is free, but you need to have Google Cloud Storage activated to put the resulting files there.

3) In the end you download your large files from GCS.