How to apply filter on the data dynamically while reading the filter conditions are available in the master file uploaded by the client

Question

We have a requirement where the client will upload the file with set of rules understandable by the Abinitio. We have to read the rules from the file and need to apply on the dataset dynamically.

I have tried uploading the user rules into a table and joined with the data file to pull the required columns along with the filter conditions and purl the column name in the filter by expression but while executing I am getting too long for conversion error.

Answer 1

Two Options:

1. Use lookup file instead of joining with table(upload rules to a file instead of DB, this will reduce cost of connecting with table
2. You could also dump the ruleset in a regular file and try to do cat in Pset of the graph.