How to capture the first gesture over a widget in Flutter

Question

I'm developing a mobile app with Flutter using these two packages: sliding_up_panel and mapbox_gl. I would like when the user does any kind of gesture over the map, the sliding up panel closes and gives the user more space for using the map.

It's almost done using the below code but I'm in trouble with the parallax effect.

import 'package:flutter/material.dart';
import 'package:sliding_up_panel/sliding_up_panel.dart';
import 'package:mapbox_gl/mapbox_gl.dart';

void main() {
  runApp(const HomePage());
}

class HomePage extends StatefulWidget {
  const HomePage({Key? key}) : super(key: key);

  @override
  State<HomePage> createState() => _HomePageState();
}

class _HomePageState extends State<HomePage> {
  final PanelController _panelController = PanelController();

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: "MyApp",
      home: Scaffold(
        body: SlidingUpPanel(
          controller: _panelController,
          defaultPanelState: PanelState.OPEN,
          parallaxEnabled: true,
          parallaxOffset: .5,
          panel: const Placeholder(),
          body: MapboxMap(
            accessToken: const String.fromEnvironment("ACCESS_TOKEN"),
            initialCameraPosition: const CameraPosition(
              target: LatLng(40.4165, -3.70256),
              zoom: 15,
            ),
            onMapCreated: (controller) {
              controller.addListener(() {
                if(_panelController.isPanelOpen && controller.isCameraMoving) {
                  _panelController.close();
                }
              });
            },
          ),
        ),
      ),
    );
  }
}

So, when the user for example start panning the map, the sliding up panel closes and if the user release it quickly, all works ok, like in the next GIF:

But, if the user holds down his finger, the map shakes like in this GIF:

Also, the current code doesn't detect user touches, it only works when scrolling the map.

I'm thinking the best solution would be capturing the first gesture over the map, disabling temporally it's interactivity using some wrapper until it comes to full screen, but I don't know how to do it. I tried wrapping it with the built in widget IgnorePointer() but it disables all the input and there's no way to know when the user is tapping the map for trigger the animation and later unlock the map.

Will appreciate a method to reach this idea or any other solutions that could work.

Thanks in advance!