How can use two different type of data (RGB and NIR images) in two different backbone in MaskRCNN architecture as an multi-modality approach?

Question

I am working on a project which required two different data (RGB and NIR) images as an input. And i have to trained the with two different backbone inside MaskRCNN. So how can we do that?

I tried defining two different backbone ResNet50 and ResNet101, but only one of it is used for traing the model, but i need to trained both the backbone network simultaniously so that i can used their feature for fusion before feeding to the FPN network.