We want to be able to make our app available to some users on short notice. Hence, we want to add them as iTunes Connect users and make them internal users of the build we've uploaded.
Is it possible to remove these users immediately after they've installed the build? Will it affect the installation on their phone?
Are there any other options without having to go through beta review, i.e. quick options?
Just go through the beta review. It generally only take a few hours for the first version and then if you don't have significant changes all future ones go through immediately. I have been using it with external testers and it has worked really well for quick turn around.