Hive materilize table- java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory

49 Views Asked by At

I am trying to create a meterialized view for Druid storage handler from Hive.

Steps:

CREATE TABLE employee_trans2 (
`__time` TIMESTAMP,
 id int,
 name string,
 age int,
 gender string)
 STORED AS ORC
 TBLPROPERTIES ('transactional'='true',"orc.compress"="NONE");

Insert data

  insert into employee_trans2 values(CURRENT_TIMESTAMP(),10,'basa',30,'M');

Select data

select `__time`,id,name,age,gender from default.employee_trans1;
+--------------------------+-----+--------+------+---------+
|          __time          | id  |  name  | age  | gender  |
+--------------------------+-----+--------+------+---------+
| 2023-10-30 22:43:35.534  | 10  | basa   | 30   | M       |
| 2023-10-30 22:44:29.56   | 10  | basa   | 30   | M       |
| 2023-10-30 22:44:29.56   | 20  | basa2  | 20   | M       |
| 2023-10-30 22:44:29.56   | 30  | basa3  | 33   | M       |
+--------------------------+-----+--------+------+---------+

Create meterialize table

CREATE MATERIALIZED VIEW mat_emp_hive_druid  STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler' AS select `__time`, id,name,age,gender from default.employee_trans1;

Error


INFO  : Completed executing command(queryId=hive_20231030232047_32c5e387-f582-49ac-b4e9-6581997dff07); Time taken: 29.552 seconds
Error: Error while compiling statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Reducer 2, vertexId=vertex_1697781224687_0009_3_01, diagnostics=[Task failed, taskId=task_1697781224687_0009_3_01_000001, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1697781224687_0009_3_01_000001_0:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory

But I am able to see the jar from hive lib


root@odp01:/usr/odp/3.2.2.0-1/hive# grep -irn "net/jpountz/lz4/LZ4Factory" /usr/odp/3.2.2.0-1/
Binary file /usr/odp/3.2.2.0-1/hive/lib/lz4-java-1.7.1.jar matches
Binary file /usr/odp/3.2.2.0-1/oozie/share/lib/hive/lz4-java-1.7.1.jar matches
Binary file /usr/odp/3.2.2.0-1/oozie/share/lib/spark/lz4-java-1.4.0.jar matches
Binary file /usr/odp/3.2.2.0-1/oozie/embedded-oozie-server/webapp/WEB-INF/lib/lz4-java-1.7.1.jar matches
Binary file /usr/odp/3.2.2.0-1/druid/lib/lz4-java-1.8.0.jar matches
Binary file /usr/odp/3.2.2.0-1/druid/extensions/druid-protobuf-extensions/druid-protobuf-extensions-27.0.0.3.2.2.0-1.jar matches
Binary file /usr/odp/3.2.2.0-1/druid/extensions/druid-kafka-extraction-namespace/lz4-java-1.8.0.jar matches
Binary file /usr/odp/3.2.2.0-1/druid/extensions/druid-kafka-indexing-service/lz4-java-1.8.0.jar matches
Binary file /usr/odp/3.2.2.0-1/druid/extensions/druid-avro-extensions/lz4-java-1.8.0.jar matches
Binary file /usr/odp/3.2.2.0-1/kafka/libs/lz4-java-1.7.1.jar matches
Binary file /usr/odp/3.2.2.0-1/hadoop-mapreduce/lz4-java-1.7.1.jar matches

tried to create using NONE compression format, SNAPPY compression format, but still getting the error as

java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory

Any help would be appreciated.

tried to create using NONE compression format, SNAPPY compression format, but still getting the error as

java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory

Any help would be appreciated.

0

There are 0 best solutions below