Keyboard pushes layout below statusbar (or even further up)

Question

I have the following xml

<RelativeLayout
    ...>

    <Toolbar
        .../>

    <ScrollView
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:layout_below="@id/toolbar"
        android:fitsSystemWindows="true"
        android:fillViewport="true">

        <LinearLayout
            android:layout_width="match_parent"
            android:layout_height="wrap_content"
            android:orientation="vertical">

            <TextInputLayout
                android:layout_width="match_parent"
                android:layout_height="wrap_content">

                <EditText
                    android:layout_width="match_parent"
                    android:layout_height="wrap_content"/>
            </TextInputLayout>

            <TextInputLayout
                android:layout_width="match_parent"
                android:layout_height="wrap_content">

                <EditText
                    android:layout_width="match_parent"
                    android:layout_height="wrap_content"/>
            </TextInputLayout>

            <Button
                android:layout_width="match_parent"
                android:layout_height="wrap_content"/>

        </LinearLayout>
    </ScrollView>
</RelativeLayout>

In my AndroidManifest.xml I set the windowSoftInputMode to adjustResize. But now when I click in one of those EditText fields the layout moves up and the actual field where I want to type is not even visible anymore. The scrolling works, but only downwards and not fully up. Any ideas on how to fix this? I don't want to use adjustPan because that disables the scroll in general.