Activity gets restarted after Biometric Authentication

Question

       val promptInfo = BiometricPrompt.PromptInfo.Builder()
                        .setTitle("Biometric authentication")
                        .setSubtitle("Verify using your biometric credential")
                        .setAllowedAuthenticators(BiometricManager.Authenticators.BIOMETRIC_STRONG or BiometricManager.Authenticators.DEVICE_CREDENTIAL or BiometricManager.Authenticators.BIOMETRIC_WEAK)
                        .setConfirmationRequired(false)
                        .build()

                    val biometricPrompt = BiometricPrompt(this@Fragment,ContextCompat.getMainExecutor(requireActivity()),object :
                        BiometricPrompt.AuthenticationCallback() {
                        override fun onAuthenticationSucceeded(result: BiometricPrompt.AuthenticationResult) {
                            super.onAuthenticationSucceeded(result)
                            goToNextScreen() // Commits another fragment
                        }

                        override fun onAuthenticationError(errorCode: Int, errString: CharSequence) {
                            super.onAuthenticationError(errorCode, errString)
                            Toast.makeText(context, errString, Toast.LENGTH_SHORT).show()
                        }

                        override fun onAuthenticationFailed() {
                            Toast.makeText(context,"failed to verify", Toast.LENGTH_SHORT).show()
                            super.onAuthenticationFailed()
                        }
                    })
                    biometricPrompt.authenticate(promptInfo)

I am using this above set of lines of code to verify User's biometric before taking him to the next fragment. It works perfectly fine for most devices , BUT in certain devices ( mostly Android 10 ) i get this error : Can not perform this action after onSaveInstanceState

No clue what's causing this , but my assumption is that the biometric prompt happens outside of the app causing the activity to go into onPause , but the Biometric authentication's success callback happens before it can restore its state. Though , this is just my assumption , didn't get to see this crash by myself

Any idea what could be causing this and how i can fix it ?