Navigating Between DialogFragment & Fragment with Navigation Component

Question

In the above navigation flow, I am facing an issue while navigating from Dialog 3 to Dialog 1 stack is not getting cleared --- so when I tap the device back button from Dialog 1 then takes back to Dialog 3 instead of going to Fragment.

     <fragment
        android:id="@+id/fragment1"
        android:name="fragment"
        tools:layout="@layout/fragment1" />

     <dialog
        android:id="dialog1"
        android:name="dialog1"
        tools:layout="@layout/dialog1" />

     <dialog
        android:id="@+id/dialog2"
        android:name="dialog2"
        tools:layout="@layout/dialog2" />

    <dialog
        android:id="@+id/dialog3"
        android:name="dialog3"
        tools:layout="@layout/dialog3">

        <action
            android:id="@+id/actionToDialog1"
            app:destination="@id/dialog1"
            app:launchSingleTop="true"
            app:popUpTo="@id/dialog1"
            app:popUpToInclusive="true" />

    </dialog>

On Device Back Button Tap I am having below code

navController.navigate(R.id.actionToDialog1);

FYI, navigation works fine, but once you land on Dialog 1 stack is not getting cleared and it should go to Fragment instead goes to Dialog 3