I am trying to configure IIDR CDC from source DB2 (Unix) to target Hadoop. I have installed and created instance on CDC and installed AccessServer, however, i am not able to extract the data from DB2 to non-RDBMS DB. i need help to configure instance, schema and table to extract the DB.
Name Server Port Database Schema Status
iidrcdc 10901 CARD UMT running
On
I think what you are needing is more details on setting up the Subscriptions to Hadoop using WebHDFS, but additionally - steps in managing the Kerberos ticket cache. (IBM IIDR slide deck on configuring IIDR with WebHDFS for Haoop: https://ibm.ent.box.com/s/w9yxxqi4vs2cfd7v83321hoxgvc7xr9n )
Secondly, you can change the field delimiter in the output flat file, but this requires creating a Custom User Exit class, which is very minor changes to the Sample class for the SampleDataFormatForWebHdfs.java
This link would provide more details on compiling the sample user Exits included with the IIDR product install. https://www.ibm.com/docs/en/idr/11.4.0?topic=cdfcreid-compiling-sample-java-class-user-exits-unix-linux
SampleDataFormatForWebHdfs.java should be used when targeting Hadoop, vs the basic class SampleDataStageDataFormat.java as there are some minor differences with quote escaping string values. Both the Field and row delimiters are specified by variables within the class and can be customized by compiling your own custom changes to the class and providing that in the Table mapping details panel for the User Exit class to use.
Chris Schreiber IBM CDC / IIDR Team
IBM IIDR ( InfoSphere Data Replication) has different Engines for various platforms. They interact with a specific platform/database and require a specific configuration.
You can use the same product installer from Fix Central to install the DB2 Engine, and Also the IIDR for DataStage engine ( which provides options to generate data in Hadoop through WebHDFS API Calls, in addition to flatfiles on local disk for DataStage). This will require selecting a separate Engine to install to a SEPARATE second installation path. ** you can only have one set of instances/platform per install path of IIDR, but you can have multiple engines installed on the same host. **
The IIDR community wiki page URL has changed but you can find it here : https://www.ibm.com/support/pages/ibm-data-replication-community-wiki#Hadoop along with a slide deck outlining the steps to configure a subscrpition to target Hadoop. (https://ibm.ent.box.com/s/w9yxxqi4vs2cfd7v83321hoxgvc7xr9n )
You may need to be aware of authentication setup for your Hadoop environment. If Kerberos authentication is needed, you may need to identify the principal you are going to use and configure the keys.
With the Target Engine installed and instance created, you can use the Management Console GUI tool to configure your 'Subscription' mapping from DB2 to Hadoop. (https://www.ibm.com/support/knowledgecenter/SSTRGZ_11.4.0/com.ibm.cdcdoc.mcadminguide.doc/concepts/map_to_hadoop.html )
Chris Schreiber, IBM IIDR / CDC Team