What is a good design pattern to implement a dynamic data importer tool?

Question

We are planning to build a dynamic data import tool. Basically taking information on one end in a specified format (access, excel, csv) and upload it into an web service.

The situation is that we do not know the export field names, so the application will need to be able to see the wsdl definition and map to the valid entries in the other end.

In the import section we can define most of the fields, but usually they have a few that are custom. Which I see no problem with that.

I just wonder if there is a design pattern that will fit this type of application or help with the development of it.

Answer 1

I am not sure where the complexity is in your application, so I will just give an example of how I have used patterns for importing data of different formats. I created a factory which takes file format as argument and returns a parser for particular file format. Then I use the builder pattern. The parser is provided with a builder which the parser calls as it is parsing the file to construct desired data objects in application.

// In this example file format describes a house (complex data object)
AbstractReader reader = factory.createReader("name of file format");
AbstractBuilder builder = new HouseBuilder(list_of_houses);
reader.import(text_stream, builder);

// now the list_of_houses should contain an extra house
// as defined in the text_stream

Answer 2

Probably Bridge could fit, since you have to deal with different file formats. And Façade to simplify the usage. Handle my reply with care, I'm just learning design patterns :)

Answer 3

You will probably also need Abstract Factory and Command patterns.

If the data doesn't match the input format you will probably need to transform it somehow. That's where the command pattern come in. Because the formats are dynamic, you will need to base the commands you generate off of the input. That's where Abstract factory is useful.

Answer 4

I would say the Adaptor Pattern, as you are "adapting" the data from a file to an object, like the SqlDataDataAdapter does it from a Sql table to a DataTable

have a different Adaptor for each file type/format? example SqlDataAdptor, MySqlDataAdapter, they handle the same commands but different datasources, to achive the same output DataTable

Adaptor pattern

HTH

Bones

Answer 5

Our situation is that we need to import parametric shapes from competitors files. The layout of their screen and data fields are similar but different enough so that there is a conversion process. In addition we have over a half dozen competitor and maintenance would be a nightmare if done through code only. Since most of them use tables to store their parameters for their shapes we wrote a general purpose collection of objects to convert X into Y.

In my CAD/CAM application the file import is a Command. However the conversion magic is done by a Ruleset via the following steps.

1. Import the data into a table. The field names are pulled in as well depending on the format.
2. We pass the table to a RuleSet. I will explain the structure the ruleset in a minute.
3. The Ruleset transform the data into a new set of objects (or tables) which we retrieve
4. We pass the result to the rest of the software.

A RuleSet is comprise of set of Rules. A Rule can contain another Rule. A rule has a CONDITION that it tests, and a MAP TABLE.

The MAP TABLE maps the incoming field with a field (or property) in the result. There are can be one mapping or a multitude. The mapping doesn't have to involve just poking the input value into a output field. We have a syntax for calculation and string concatenation as well.

This syntax is also used in the Condition and can incorporate multiple files like ([INFIELD1] & "-" & [INFIELD2])="A-B" or [DIM1] + [DIM2] > 10. Anything between the brackets is substituted with a incoming field.

Rules can contain other Rules. The way this works is that in order for a sub Rule mapping to apply both it's condition and those of it's parent (or parents) have to be true. If a subRule has a mapping that conflicts with a parent's mapping then the subRule Mapping applies.

If two Rules on the same level have condition that are true and have conflicting mapping then the rule with the higher index (or lower on the list if you are looking at tree view) will have it's mapping apply.

Nested Rules is equivalent to ANDs while rules on the same level are equivalent of ORs.

The result is a mapping table that is applied to the incoming data to transform it to the needed output.

It is amicable to be being displayed in a UI. Namely a Treeview showing the rules hierarchy and a side panel showing the mapping table and conditions of the rule. Just as importantly you can create wizards that automate common rule structures.