BizTalk Flat File complexity parsing issue

Question

I'm currently creating a flat file schema to implement an old UK EDI format called Tradacoms. I've replicated what I need for the part of the schema I'm dealing with and it generally works fine. However because there are lots of optional items in the schema I need to change the Parser Optimisation to Complexity.

To easily explain the problem I've reproduced the issue to a much smaller schema (not related to Tradacoms at all in fact).

<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns="http://Bidvest.Integration.Supplier.Schemas.TestSchema" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" targetNamespace="http://Bidvest.Integration.Supplier.Schemas.TestSchema" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:annotation>
    <xs:appinfo>
      <b:schemaInfo standard="Flat File" root_reference="Root" default_pad_char=" " pad_char_type="char" count_positions_by_byte="false" parser_optimization="complexity" lookahead_depth="0" suppress_empty_nodes="false" generate_empty_nodes="true" allow_early_termination="false" early_terminate_optional_fields="false" allow_message_breakup_of_infix_root="false" compile_parse_tables="false" />
      <schemaEditorExtension:schemaInfo namespaceAlias="b" extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension" standardName="Flat File" xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />
    </xs:appinfo>
  </xs:annotation>
  <xs:element name="Root">
    <xs:annotation>
      <xs:appinfo>
        <b:recordInfo structure="delimited" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" sequence_number="1" child_order="infix" child_delimiter_type="char" child_delimiter="+" />
      </xs:appinfo>
    </xs:annotation>
    <xs:complexType>
      <xs:sequence>
        <xs:annotation>
          <xs:appinfo>
            <b:groupInfo sequence_number="0" />
          </xs:appinfo>
        </xs:annotation>
        <xs:element name="Name" type="xs:string">
          <xs:annotation>
            <xs:appinfo>
              <b:fieldInfo justification="left" sequence_number="1" />
            </xs:appinfo>
          </xs:annotation>
        </xs:element>
        <xs:element minOccurs="0" name="Address">
          <xs:annotation>
            <xs:appinfo>
              <b:recordInfo sequence_number="2" structure="delimited" preserve_delimiter_for_empty_data="false" suppress_trailing_delimiters="false" child_order="infix" child_delimiter_type="char" child_delimiter=":" />
            </xs:appinfo>
          </xs:annotation>
          <xs:complexType>
            <xs:sequence>
              <xs:annotation>
                <xs:appinfo>
                  <b:groupInfo sequence_number="0" />
                </xs:appinfo>
              </xs:annotation>
              <xs:element minOccurs="0" name="Line1" type="xs:string">
                <xs:annotation>
                  <xs:appinfo>
                    <b:fieldInfo justification="left" sequence_number="1" />
                  </xs:appinfo>
                </xs:annotation>
              </xs:element>
              <xs:element minOccurs="0" name="Line2" type="xs:string">
                <xs:annotation>
                  <xs:appinfo>
                    <b:fieldInfo justification="left" sequence_number="2" />
                  </xs:appinfo>
                </xs:annotation>
              </xs:element>
              <xs:element minOccurs="0" name="Line3" type="xs:string">
                <xs:annotation>
                  <xs:appinfo>
                    <b:fieldInfo justification="left" sequence_number="3" />
                  </xs:appinfo>
                </xs:annotation>
              </xs:element>
              <xs:element minOccurs="0" name="Line4" type="xs:string">
                <xs:annotation>
                  <xs:appinfo>
                    <b:fieldInfo justification="left" sequence_number="4" />
                  </xs:appinfo>
                </xs:annotation>
              </xs:element>
              <xs:element minOccurs="0" name="Line5" type="xs:string">
                <xs:annotation>
                  <xs:appinfo>
                    <b:fieldInfo justification="left" sequence_number="5" />
                  </xs:appinfo>
                </xs:annotation>
              </xs:element>
              <xs:element minOccurs="0" name="PostCode" type="xs:string">
                <xs:annotation>
                  <xs:appinfo>
                    <b:fieldInfo justification="left" sequence_number="6" />
                  </xs:appinfo>
                </xs:annotation>
              </xs:element>
              <xs:element minOccurs="0" name="Country" type="xs:string">
                <xs:annotation>
                  <xs:appinfo>
                    <b:fieldInfo sequence_number="7" justification="left" />
                  </xs:appinfo>
                </xs:annotation>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

The schema contains a name element and an Address record which itself has a number of optional elements.

If I validate an instance (right clicking on the schema etc) with the test file below

DAve+Line1:Line2:Line3:Line4:Line5:PostCode:Country

then I get the output below as expected

<Root xmlns="http://Bidvest.Integration.Supplier.Schemas.TestSchema">
  <Name xmlns="">DAve</Name>
  <Address xmlns="">
    <Line1>Line1</Line1>
    <Line2>Line2</Line2>
    <Line3>Line3</Line3>
    <Line4>Line4</Line4>
    <Line5>Line5</Line5>
    <PostCode>PostCode</PostCode>
    <Country>Country</Country>
  </Address>
</Root>

If I validate an instance with a very simple message as below

DAve+Line1

then I get the following output

<Root xmlns="http://Bidvest.Integration.Supplier.Schemas.TestSchema">
  <Name xmlns="">DAve</Name>
  <Address xmlns="">
    <Line4>Line1</Line4>
  </Address>
</Root>

You can see that Line1 has been placed in the Line4 element. As the sample message above has the text 'Line1' as the first value before the delimiter, I would have expected the XML above to be Line1.

Something very strange is going on here. Can anyone help? I have this problem in BizTalk 2013 (CU3) and BizTalk 2013 R2.

Answer 1

Changing the Parser Optimisation back to default speed should fix your problem.

Answer 2

Yes, the Flat File disassembler can get very confused if you don't have the mandatory fields at the start of a record. You've made all of the Address elements optional and then it can get very strange results. I've found that you should always have at least one mandatory field as the first field and you should never have a mandatory field after an optional one.

If you remove the minOccurs = 0 on line1 it works correctly and you get the following.

<Root xmlns="http://Bidvest.Integration.Supplier.Schemas.TestSchema">
    <Name xmlns="">DAve</Name>
    <Address xmlns="">
        <Line1>Line1</Line1>
    </Address>
</Root>

It will even handle the following input

DAve+

which gets the following output

<Root xmlns="http://Bidvest.Integration.Supplier.Schemas.TestSchema">
    <Name xmlns="">DAve</Name>
    <Address xmlns="">
        <Line1/>
    </Address>
</Root>

Or

DAve

output

<Root xmlns="http://Bidvest.Integration.Supplier.Schemas.TestSchema">
    <Name xmlns="">DAve</Name>
</Root>