I have this issue when using duckdb with ibis in iPython:
So I use duckdb with ibis and try to access parquet data on s3.
I basically do:
import boto3
import ibis
def get_duck_db_client() -> BaseBackend:
boto3.resource("s3")
boto3.client("s3")
con = ibis.duckdb.connect(database=":memory:", read_only=False)
con.raw_sql(
f"""
INSTALL httpfs;
LOAD httpfs;
SET s3_region='{AWS_REGION}';
SET s3_access_key_id='{AWS_ACCESS_KEY}';
SET s3_secret_access_key='{AWS_SECRET_KEY}';
"""
)
return con
if __name__ == "__main__":
con = get_duck_db_client()
data = con.read_parquet(S3_PATH)
I want to use this in an ipython shell. But, when I do %run script.py I get:
OperationalError: (duckdb.duckdb.TransactionException) TransactionContext Error: cannot start a transaction within a transaction
(Background on this error at: https://sqlalche.me/e/20/e3q8)
Things work fine when I just execute the code in the ipython shell directly (i.e. without running the script).
Is there a way to prevent this in this setting?