Bigquery data not getting inserted

Question

I'm using python client library to insert data to big query table. The code is as follows.

client = bigquery.Client(project_id)
errors = client.insert_rows_json(table=tablename,json_rows=data_to_insert)
assert errors == []

There are no errors, but the data is also not getting inserted.

Sample JSON rows:

[{'a':'b','c':'d'},{'a':'f','q':'r'},.....}]

What's the problem? No exception also

Answer 1

got the answer to my question. The problem was I was inserting one more column data for which data was not there. I found a hack in order to find out if the data is not inserting to bigquery table.

1. Change the data to newline delimited json with the keys as the column names and values as values you want for that particular column.
2. bq --location=US load --source_format=NEWLINE_DELIMITED_JSON dataset.tablename newline_delimited_json_file.json. Run this command in you terminal and see if throws any errors. If it throws an error it's likely that something is wrong with your data/table schema.
3. Change the data/table schema as per the error and retry inserting the same via python.

It's better if the python API throws an error/exception like on the terminal, it would be helpful.

Answer 2

It can be 2 possible situations:

1. your data does not match the schema
2. your table is freshly created, and the update is just not yet available

References:

• Related GitHub issue
• Data availability

Answer 3

client.insert_rows_json method using StreamingInsert . Inserting data to BigQuery using StreamingInsert will be cause of latency on table preview on BigQuery console.
The data is not appeared immediately. So, You need to query them to confirm the data inserted.