Can we impose custom DAG plan over generated DAG plan in TEZ? If possible, How?

71 Views Asked by At

Instead of using a DAG plan generated by TEZ in Hive, I want to use custom DAG plan created by me. Is it Possible? If Yes, can anyone explain?

1

There are 1 best solutions below

0
On BEST ANSWER

If you are using Hive for running queries then you cannot use custom DAG. If you do low level programming Tez application using API, you can. See High level docs here: https://hortonworks.com/blog/expressing-data-processing-in-apache-tez/

Something like this:

//Define DAG
DAG dag = new DAG();
Vertex map1 = new Vertex(MapProcessor.class) 
and so on...

See apache project examples: https://git-wip-us.apache.org/repos/asf?p=tez.git;a=blob;f=tez-examples/src/main/java/org/apache/tez/examples/HashJoinExample.java