Migrate Dataflow NIFI

Question

I was using NIFI in a standalone version and now that I have installed a clustered NIFI version with HDF, i want to migrate my DataFlow from my standalone version to the clustered one .

What's the best way to do this ?

Answer 1

Assuming you have active data in your standalone NiFi, you would:

• Stop all the source processors and let your flow bleed out until there is no active data in any of the queues.
• Stop the HDF cluster
• Stop the standalone instance and copy conf/flow.xml.gz from the standalone instance to each of the nodes of the HDF cluster at /var/lib/nifi/conf/
• Start your HDF cluster

If you don't care about anything going on in your standalone NiFi, then skip the first step.

UPDATE:

An additional consideration is dealing with sensitive properties (i.e. password fields in processors)...

In nifi.properties the property nifi.sensitive.props.key is used to encrypt sensitive properties in the flow.xml.gz. You are not forced to fill in a value for this property, and if you leave it blank there is a default password in the code.

The Ambari install of HDF forces you to set a value for nifi.sensitive.props.key, so unless you set the values the same on both instances, then the HDF cluster will be trying to decrypt values in the flow.xml.gz using a different key than what they were encrypted with.

The nifi-toolkit (which can be downloaded from the main NiFi page) offers a tool to migrate the sensitive properties key for a flow.xml.gz to a new value.

Here is an example...

Lets say nifi-1 is the original NiFi instance and the sensitive properties key for nifi-1 is set to 'password1'.

Then you copied that flow.xml.gz to nifi-2 and the sensitive properties key for nifi-2 is 'password2'.

Running this command would read in the flow.xml.gz from nifi-1 using the original sensitive properties key, and write out a new version of in nifi-2 with the sensitive properties encrypted with the new password:

./nifi-toolkit-1.2.0-SNAPSHOT/bin/encrypt-config.sh -f /path/to/nifi-1/conf/flow.xml.gz -g /path/to/nifi-2/conf/flow.xml.gz -s password2 -n /path/to/nifi-1/conf/nifi.properties -o /path/to/nifi-2/conf/nifi.properties -x

• -f is the source flow.xml.gz (nifi-1)
• -g is the destination flow.xml.gz (nifi-2)
• -s is the new sensitive properties key (nifi-2)
• -n is the source nifi.properties (nifi-1)
• -o is the destination nifi.properties (nifi-2)
• -x tells it to only process the sensitive properties