I am trying to read data into a dataframe on a databricks worspace.
The data sits in a hive metastore of a different databricks workspace where i have only sql warehouse access.
The datasets are huge over 600 million records in one table.
How do i do this efficiently.
Any other alternate method to do this ?
I have tried.
Connecting through sqlalchemy and also databricks-sql-connector.
Both are working but are very slow to fetch data.