Uploading large excel files to Big Query

Question

how can i upload a large excel file and dataset (larger than 10 mb) to big query ? Can anyone help ?

I tried to research a way to do it, but everything i found was a little bit complicated. I just started working with SQL and I don't have much experience.

Answer 1

!!! WARNING prior to answer.

The below answer will incur costs. You can use --dry-run to estimate the number of bytes that are processed for bq and then combine it with Pricing calculator:

https://cloud.google.com/products/calculator/

What you are trying to achieve, is possible by loading a dataset from a google cloud storage bucket.

To make it easy for new users, there is a "Guide me" option from GCP on how to upload datasets larger than 10 mb.

GCP "Guide me" site

Don't forget to first estimate the costs.

An example of how this can be used using a Google Storage URI is referenced in this page:

  bq load \
    --source_format=CSV \
    mydataset.mytable \
    gs://mybucket/mydata.csv \
    ./myschema.json

Again, use pricing calculator to estimate costs.

To be able to create buckets on a specific project, you need specific IAM permissions, also referenced here