IllegalStateException: Restore State failed: destination Fragment2 cannot be found from the current destination Fragment4

Question

I am using the Jetpack Navigation library, Graph A is a graph similar to Login, where Graph B is the main content containing a BottomNavigationView, the problem occurs in one scenario only.

User is shown Graph A upon opening application. After some checks, using the setGraph method on the NavController, we change it Graph B to remove the back stack of Graph A.

1. user goes to the startDestination (Fragment 1) on Graph B1
2. navigate to Fragment 2 from Fragment 1
3. Click the bottom navigation to go to Graph B3 (presenting Fragment 4)
4. Click button to go to Graph A (using .setGraph)

Actual Result App crashes with

IllegalStateException: Restore State failed: destination Fragment2 cannot be found from the current destination Fragment4

Expected Result App goes to Graph A

Note: This only occurs if you navigate to a nested fragment on Graph B1. Same scenario on Graph B2 doesn't cause this (Going from fragment 2 to a fragment also on Graph B2, then to fragment 4)