Design pattern for grouping fields and structuring data in a flat file

Question

I am using the FileHelpers C# library to read a file into an array of custom objects that will be processed. For instance, a partial file definition:

[FixedLengthRecord]
public class Row
{
    [FieldFixedLength(9)]
    [FieldTrim(TrimMode.Right)]
    public string Ssn;

    [FieldFixedLength(15)]
    [FieldTrim(TrimMode.Right)]
    public string LastName;

    ...
}

I'm trying to abide by OOP principles while doing this and I've noticed that some fields have a natural grouping (e.g. SSN and ClientID, EmployerID and EmployerName, etc.), so I tried to break them up into separate classes (e.g. Client and Employer). But this seems problematic because some of the fields need to be shared across objects (e.g. the ClientID needs to know the associated EmployerID).

To further complicate things, I would like to add fields with the [FieldNotInFile] attribute to the class definition(s) because during processing, my application will query for database records that match a specific row field and populate the row's respective [FieldNotInFile]s (e.g. get client's FName from database based on SSN since it's not in the file).

How would you structure this? I kind of just want to keep the whole file row as one class, but it's approaching 75 fields, which seems ridiculous. Thoughts?

Answer 1

A FileHelpers class is just a way of defining the specification of a flat file using C# syntax.

As such, the FileHelpers classes are an unusual type of C# class and you should not try to use accepted OOP principles. (There are many ways a FileHelpers class violates OOP principles, most obviously it requires you to use public fields instead of properties). FileHelpers should not have properties or methods beyond the ones used by the FileHelpers library.

Think of the FileHelpers class as the 'specification' of your CSV format only. That should be its only role. Then if you need the records in a more 'normal' object, then map the results to something better:

FileHelperEngine engine = new FileHelperEngine<FileHelpersOrder>(); 
var records = engine.ReadFile("FileIn.txt");

var niceOrders = records.Select(
    x => new NiceOrder() 
       { Number = x.Number,  
         Customer = x.Customer 
         // etc.
       });

Where FileHelpersOrder is your CSV specification and the NiceOrder class would be a proper OOP class with properties, methods, etc. as necessary.

Answer 2

I'm not sure of a specific pattern but I would not include the extra fields in the class that's meant for deserializing/parsing your flat file. You can add the database field to a inherited class.

Answer 3

In using file-helpers, I consider the FileHelper class to represent a record in the file. The attribute you put at the top of the class (FixedLengthRecord or DelimitedRecord) reminds you of this--this is a file record, not a model of an actual entity.

This means if you need to have a object which models an entity, you would have to do a lot of copying from the FileHelper class into your entity class.

I have many FileHelper classes with around 75 fields--each of these classes represent a file specification, which is often large. But these are still very maintainable--simply open the spec and code file side by side and go down the lists of the field to find any changes you need to make.

I would however, split up classes for headers, trailers, and different row types. Then I would use the MultiRecord engine to decide which class to use.