how to fix "field units already exist in schema" for pandas gpq

Question

Versions: Mac OS Mojave 10.14.5 Python 3.6.5 Pandas 0.24.2 pandas-gbq 0.10.0

I am trying to pull data from the shipstation api and load it into bigquery to use in our BI platform (tableau). I have successfully made the api call, which returned a json object. I have successfully turned that in a pandas df object. When I try to write that df to the dataset in BigQuery, I get the error: "field units already exist in schema", even though the table i'm writing doesn't exist and is created by pandas-gbq.

I have tried to create the table i'm writing to and just append fields, but I get a similar error that I need to make sure my schemas are aligned. I have tried setting if_exists=replace, which also didn't work. I found a few questions that suggest making sure all my columns are timestamp objects and not datetime, which didn't fix the issue either.

Here is my code:

df = pd.io.json.json_normalize(data)
df.columns = df.columns.map(lambda x: x.split(".")[-1])
pandas_gbq.to_gbq(df, 'my_dataset.my_table', project_id='project-id',if_exists='replace')

Here is the error:

GenericGBQException: Reason: 400 POST https://www.googleapis.com/bigquery/v2/projects/forte-data/datasets/ss/tables: Field units already exists in schema

Any help would be greatly appreciated.

thanks

Answer 1

Hey maybe late but run into a similar problem,

Probably the column is duplicated in your DF, check for "field" (lower case) in your DF