Data Transformation for Large data in a file

Question

I am new to ensemble and have a clarification regarding the Data Transformations.

I have 2 schemas as follows,

PatientID,
Patient Name,
Patient Address (combination of door number, Street, District, State)

and another schema as,

PatientID,
Patient Name,
Door Number
Street
District
State

Now there is an incoming text file with 1000's of records as per the first schema ('|' separated) as below,

1001|John|220,W Maude Ave,Suisun City, CA

like this there a 1000's of recrods in the input file

My requirement is to convert this as per the second schema (i.e to separate the Address) and store in the file like,

1001|John|220|W Maude Ave|Suisun City|CA

One solution I implemented was to loop through each line in the file and replace the , in the address with '|'.

My question is, whether we can do it through DTL. If the answer is yes how do we loop through 1000s of records using DTL.

Whether DTL will be time consuming? because we need to load the schema and then do the transformations.

Please help.

Answer 1

You can use DTL with any class that inherit from Ens.VirtualDocument or %XML.Adaptor, virtually Ensemble use class dictionary to represent the schema so for basic classes there is not problem is you extends %XML.Adaptor Ensemble can represent it. In case of virtual documents the object has to be set the DocType.

In order to do the loop there is a in DTL

Answer 2

Yes, DTLs can parse 1000's of records. You can do the following: 1) Create a record map to parse the incoming file that has schema 1 2) Define an intermediate object that maps schema 2 fields to object properties 3) Create a DTL whose source object is the record map object from 1 above and target is object from 2 above.