Where to create staging data table in BigData environment?

Question

I am currently having Hadoop-2, PIG, HIVE and HBASE. I have an inputdata. I have loaded that data in HDFS. I want to create staging data in this environment.

My query is -

In which BigData component, I should create Staging Table(Pig/HIVE/HBASE) ; this will have data coming in based on a condition? Later, we might want to run MapReduce Jobs with complex logic on it.

Please assist

Answer 1

Hive: If you have OLAP kind of workload and dont need realtime read/write.

HBase: If you have OLTP kind of workload. You need to do realtime/streaming read/write. Some batch or OLAP processing can be done by using MapReduce. SQL-like querying is possible by using Apache Phoenix.

You can run MapReduce job on HIVE and HBase both.

Answer 2

Anywhere you want. Pig is not an option as it does not have a metastore. Hive if you want SQL Like queries. HBase based on your access patterns.

When you run a Hive query on top of data it is converted into MR.

When you create it in Hive use Hive Queries & not MR. If you are using MR then use Pig. You will not benefit creating a Hive table on top of data.