Touch events and gestures priority Bottom sheet

Question

I have a ModalBottomSheetLayout and in its content, I have a LazyVerticalGrid, the user can interact with the bottom sheet by swiping to expand, half expand, and hide.

In my case whenever I scroll the LazyVerticalGrid while the bottom sheet is half expanded, the bottom sheet expands first before allowing me to scroll, which is not the desired behavior.

I want the behavior to be exactly as the comments bottom sheet in Instagram, you can scroll all over the comments and you can still interact with the bottom sheet from Hadler or whatsoever.

here is my code

val modalSheetState = rememberModalBottomSheetState(
    initialValue = ModalBottomSheetValue.Hidden,
    skipHalfExpanded = false,
)
ModalBottomSheetLayout(
//    modifier = modifier.focusProperties { canFocus = false },
    sheetState = modalSheetState
){ ...
    LazyVerticalGrid(
        modifier = modifier
            .fillMaxSize(),
//            .focusRequester(FocusRequester.Default)
//            .focusable(enabled = true)
//            .focusGroup()
//            .focusTarget()
            .pointerInput(Unit) {
                detectDragGestures { change, dragAmount ->
                    corutineScope.launch {
                        rememberLazyGridState.animateScrollBy(
                            -1 * dragAmount * 25, animationSpec = spring(
                                dampingRatio = Spring.DampingRatioNoBouncy,
                                stiffness = Spring.StiffnessVeryLow
                            )
                        )
                    }
                    Log.e("grid", "drag")
                }
            },
        userScrollEnabled = true /*false*/, state = rememberLazyGridState, columns = GridCells.Fixed(4)
    ) {
        items(...)
      }
}

with that code above if the drag gesture is horizontal the message is logged, and the touch is consumed and doesn't get through to the bottom sheet which is what I want, however, if the drag is vertical the touch is consumed in the bottom sheet first and the message is not logged.

I could kinda achieve the desired behavior by setting userScrollEnabled to false and implementing my own scrolling, but it doesn't feel natural still missing the dragging and the fling. Am I heading in the right direction?

I tried playing around in focus, but nothing happened.

Any help would be appreciated