How copy iceberg table with piece of data into local hive?

33 Views Asked by At

What is the problem

Iceberg tables have a definition in hive that point to parquet files on s3. So there is unclear how copy table definition to local hive that will point to couple of data files.

Problem 1

With org.apache.hadoop.hive.metastore.HiveMetaStoreClient I can list, get tables from the poduction hive and than create the same tables in the local one. But copied table from production will point to metadata that does not exist locally. Also metadata points to real files that can be very big. So even if I copy all metadata - the table will not work because locally it points to non-existing data.

Problem 2

For a simple parquet table, I can create a table, and download a couple of files locally and the table will work. With Iceberg, downloading files does not help because they are not present in metadata. Is there a way to "connect" these files to the existing hive table without crating Trino jdbc connection and perform batch inserts from existing table to the new one?

0

There are 0 best solutions below