BizTalk FF Disassembler - Only Working With Very Small Files

Question

I have a BizTalk 2016 / VS 2015 Solution. There's a simple flat file schema that debatches each row by having a repeating record element have its "Max Occurs" property set to 0.

The "config" part of the schema can be seen below:

<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns="http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" attributeFormDefault="unqualified" elementFormDefault="qualified" targetNamespace="http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:annotation>
    <xs:appinfo>
      <schemaEditorExtension:schemaInfo namespaceAlias="b" extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension" standardName="Flat File" xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />
      <b:schemaInfo standard="Flat File" codepage="65001" default_pad_char=" " pad_char_type="char" count_positions_by_byte="false" parser_optimization="speed" lookahead_depth="3" suppress_empty_nodes="false" generate_empty_nodes="true" allow_early_termination="true" early_terminate_optional_fields="true" allow_message_breakup_of_infix_root="true" compile_parse_tables="false" root_reference="Employee" />
    </xs:appinfo>
  </xs:annotation>
  <xs:element name="Employee">
    <xs:annotation>
      <xs:appinfo>
        <b:recordInfo structure="delimited" child_delimiter_type="hex" child_delimiter="0xA" child_order="postfix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
        <b:properties>
          <b:property distinguished="true" xpath="/*[local-name()='Employee' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']/*[local-name()='Data' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']/*[local-name()='NiNumber' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']" />
          <b:property distinguished="true" xpath="/*[local-name()='Employee' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']/*[local-name()='Data' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']/*[local-name()='OrgLeaveDate' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']" />
          <b:property distinguished="true" xpath="/*[local-name()='Employee' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']/*[local-name()='Data' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']/*[local-name()='PosLeaveDate' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']" />
          <b:property distinguished="true" xpath="/*[local-name()='Employee' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']/*[local-name()='Data' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']/*[local-name()='PostStartDate' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']" />
        </b:properties>
      </xs:appinfo>
    </xs:annotation>
    <xs:complexType>
      <xs:sequence>
        <xs:annotation>
          <xs:appinfo>
            <groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
          </xs:appinfo>
        </xs:annotation>
        <xs:element maxOccurs="1" name="Data">
          <xs:annotation>
            <xs:appinfo>
              <b:recordInfo structure="delimited" child_delimiter_type="hex" child_delimiter="0x9" child_order="infix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
            </xs:appinfo>
          </xs:annotation>
          <xs:complexType>

The input file is tab delimited with unix style line endings.

The only slightly unusual thing about the flat-file schema is that the last element "PostStartDate" has a micOccurs attribute value of 0. This is because there are a couple of providers of these document and one of the providers doesn't include a "PostStartDate" column.

I've setup a receive location that contains a FF disasembler. It has the DocumentSpecName property set to that of the schema. All other properties are as default.

The only subscriber to this receive is a send port that sends to a local folder.

If I drop a flat file with 3 rows then all works ok. However, if I extend with a few more rows (7 total in this example) then BizTalk suspends on the receive location with the error: "The Messaging Engine encountered an error during the processing of one or more inbound messages." In the event log, I see the error:

There was a failure executing the receive pipeline: "LCC.Integration.Common.Pipelines.RcvArchiveFF, LCC.Integration.Common.Pipelines, Version=1.0.0.0, Culture=neutral, PublicKeyToken=ff3873c2d0a7e656" Source: "Flat file disassembler" Receive Port: "SharePoint.Employees.Receive.SAPExtract" URI: "D:\FileDrop\Employees\In\*.txt" Reason: Unexpected data found while looking for:
'\t'
The current definition being parsed is Employee. The stream offset where the error occured is 1813. The line number where the error occured is 8. The column where the error occured is 0.

Another thing I've noticed, if I remove the trailing LF from the last row in the file then it works - regardless of the number of rows.

I've made a screencast to demo the problem: Screencast of Problem

I've created a sample solution to demonstrate the problem. This is available at the following github repo: https://github.com/RobBowman/BizTalkFFProblem

Any ideas?

Answer 1

Edit: I just tried your GitHub Sample in BizTalk 2013R2 and it's working fine without the change.

Edit2: Broken when >20 rows.

Edit3: I now realize that it's not really supported, the setting is: 'Allow Message Breakup At Infix Root'... your schema is not infix, it's postfix. Debatching FlatFiles on the disassember is only supported when infix. If possible, you should make your schema to support infix, then it'll work.