Android Chrome Custom Tab Intent Filter Restricted to Only My App?

Question

I'm trying to achieve https://mobikul.com/use-safariviewcontroller-callback-url-objective-c/ on Android, where "sourceApplication" is checked in the iOS example and only allows an SFSafariViewController to redirect back to the application (presumably it's a SFSafariViewController instantiated by the app...but that's another question!).

I am having trouble achieving parity in Android. In my app, BrowserActivity opens up a Chrome Custom Tab, which redirects back to the CallbackActivity from the HTML. Chrome / any browser on phone, I find, is also allowed to redirect back to CallbackActivity if I just enter the correct URL schema, and that's bad news! It seems like there should be a whitelist mechanism in the manifest to only allow a Chrome Custom Tab opened from my application to redirect back to my activity, while disallowing anybody else. I can not set android:exported="false" on CallbackActivity in the manifest because that breaks the redirect capability from my application. How does one achieve this?

Here's what I've got for the Android manifest:

    <activity android:name="com.epicgames.ue4.SplashActivity" android:launchMode="singleTask" android:debuggable="true">
      <intent-filter>
        <action android:name="android.intent.action.MAIN" />
        <category android:name="android.intent.category.LAUNCHER" />
      </intent-filter>
    </activity>
    <activity android:name="com.epicgames.ue4.GameActivity" android:exported="false"> 
      <meta-data android:name="android.app.lib_name" android:value="UE4" />
    </activity>
    <activity android:name=".BrowserActivity" android:launchMode="singleTask" android:exported="false">
      <intent-filter>
        <action android:name="android.intent.action.VIEW" />
        <category android:name="android.intent.category.DEFAULT" />
        <category android:name="android.intent.category.BROWSABLE" />
      </intent-filter>
    </activity>
    <activity android:name=".CallbackActivity" android:launchMode="singleTask">
      <intent-filter>
        <action android:name="android.intent.action.VIEW" />
        <category android:name="android.intent.category.DEFAULT" />
        <category android:name="android.intent.category.BROWSABLE" />
        <data android:scheme="customscheme" android:host="callback" />
      </intent-filter>

And here's what I've got for the relevant activities:

public class BrowserActivity extends Activity {
    @Override
    public void onCreate(Bundle bundle) {
        super.onCreate(bundle);

        final Uri uri = Uri.parse(getIntent().getStringExtra("url"));

        final CustomTabsIntent intent = new CustomTabsIntent.Builder().build();
        intent.intent.addFlags(Intent.FLAG_ACTIVITY_NO_HISTORY);
        intent.intent.addFlags(Intent.FLAG_ACTIVITY_CLEAR_TOP);
        intent.intent.addFlags(Intent.FLAG_ACTIVITY_SINGLE_TOP);
        intent.intent.setData(uri);

        startActivity(intent.intent);
    }
}

public class CallbackActivity extends Activity {
    @Override
    public void onCreate(Bundle bundle) {
        super.onCreate(bundle);

        //TODO validate only chrome custom tab HTML, started from my app, redirected here
        //getCallingPackage() is always null and referrer host can be spoofed...how to?

        final Intent intent = new Intent(this, com.epicgames.ue4.GameActivity.class);
        intent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK|Intent.FLAG_ACTIVITY_SINGLE_TOP);
 
        startActivity(intent);
    }
}