Android system audio not working correctly after using setSpeakerPhoneOn of AudioManager

Question

I'm trying to make a application that switches the audio play between the speaker and earpiece. I'm using The following code:

 public boolean setAudioMode(String mode) {
    AudioManager audioManager =
            (AudioManager) this.getSystemService(Context.AUDIO_SERVICE);
    int currentVolume = audioManager.getStreamVolume(AudioManager.STREAM_MUSIC);
    audioManager.setStreamVolume(AudioManager.STREAM_MUSIC, currentVolume, 0);

    if (mode.equals("earpiece")) {
        audioManager.setMode(AudioManager.STREAM_MUSIC);
        audioManager.setSpeakerphoneOn(false);
        this.setVolumeControlStream(AudioManager.STREAM_MUSIC);
        return true;
    } else if (mode.equals("speaker")) {
        audioManager.setMode(AudioManager.STREAM_MUSIC);
        audioManager.setSpeakerphoneOn(true);
        this.setVolumeControlStream(AudioManager.STREAM_MUSIC);
        return true;
    }

    return false;
}

This code is working and I can change the audio output correctly. But after I close my application, my phone does not play any sound through my headphones and the volume control sticks to in call volume. My phone only go back to normal after a full restart. What can I do to restore the phone audio? Or is there a better way to switch the audio output that does not make this problem?