Flutter Google Maps cannot drag the map with nested navigator

Question

i am stuck about 6 days until now about how to drag the google maps using stack and nested navigator, below is the screenshot and code using navigator :

Image Using Nested Navigator

@override
  Widget build(BuildContext context) {
    return Scaffold(
        backgroundColor: Colors.white,
        body: Stack(
          children: [
            GoogleMap(
                mapType: MapType.normal,
                initialCameraPosition: CameraPosition(
                  target: LatLng(-6.168033, 106.900467),
                  zoom: 14,
                ),
                onMapCreated: (GoogleMapController controller) {

                },
                onTap: (position) {
                  print('tap map');
                }
            ),
            Navigator(
              onGenerateRoute: (settings) {
                return PageRouteBuilder(
                    opaque: false,
                    transitionDuration: Duration(milliseconds: 0),
                    reverseTransitionDuration: Duration(milliseconds: 0),
                    pageBuilder: (BuildContext context, _, __) {
                      return Center(child: Text('you cannot drag the maps outside this text'));
                    }
                  );
              },
            )
          ],
        )
    );
  }

If i just using a container and not navigator, google maps can be dragged like in the below screenshot :

Image Not Using Nested Navigator

@override
  Widget build(BuildContext context) {
    return Scaffold(
        backgroundColor: Colors.white,
        body: Stack(
          children: [
            GoogleMap(
                mapType: MapType.normal,
                initialCameraPosition: CameraPosition(
                  target: LatLng(-6.168033, 106.900467),
                  zoom: 14,
                ),
                onMapCreated: (GoogleMapController controller) {

                },
                onTap: (position) {
                  print('tap map');
                }
            ),
            Container(
              child: Center(child: Text('you can drag the maps outside this text')),
            ),
          ],
        )
    );
  }

What i want to achive is to use nested navigator, so i can move from page/screen to another page/screen and using the google maps in the background stack.

Actualy i try to wrap Navigator with IgnorePointer and the maps can be dragged, but IgnorePointer will ignored a whole screen so i cannot use that because in every screen on NestedNavigator will have some content like button, sliding up panel, bottomsheet, etc.

Anyone can help me ?

Thank you for any help