Detecting incorrect custom model - Compose model in Form Recognizer

Question

I have created a compose model with multiple custom models. For one specific document where it was supposed to use model A for analysis, ends up using model B which is incorrect.

Here are few specific to know:

• Applications could be completely filled or partially filled (So I added both to my training set)
• Both the models in discussion are trained on application form totally different in format

Observations:

• If the test document is tested on model A(correct model), the confidence is 80%
• If the test document is tested on compose model, modelB is taken up for analyze and docType Confidence is 21% in this case

Q1) Isn't compose model supposed to perform analyze with best fit model? In this case model A is best fit. But model B is selected.

Q2) What's the best practices for training when composing huge set of models (say in the range of 40)

Q3) How can issues like this be fixed?

Answer 1

I received a response from Microsoft team on this, The way compose model works seems to be refined on 3.0 version. I tried with exact same training data on Form Recognizer Studio (3.0) and the compose feature is working as expected!

Answer 2

Composed model selects the best fit model to analyze the document. Are documents in Model A and Model B similar in structure ? If the documents are similar you might get higher accuracy creating a single model for both documents. Try adding documents from model B into the training set of model A and compose without Model B and see if that improves the accuracy of the composed model.