SwiftUI transmit drag focus to outer view (ScrollView inside dragable)

Question

What I want

I have a "ScrollView" in a custom sheet that I can drag down to close. Of course, the ScrollView is above the drag area, so when I drag over the ScrollView it scrolls down or up. I want to disable the ScrollView when I am at the top of the ScrollView and scroll up so that the sheet is dragged and starts to close. This is similar to the behaviour of the Shazam sheet.

The Problem

If the ScrollView is deactivated, the current drag action is not applied to the sheet, but does nothing. Only when dragging again (on the now deactivated ScrollView) the sheet is focused on the dragging. So is there a way to transfer the focus of the drag action from the ScrollView to the outer sheet view without starting a new one?

Showcase

I created a simplified version of the problem for better understanding.

We have a container (inner) that is draggable along the y-axis. On drag release, we let it jump back to offset 0.

@State var offsetY: CGFloat = .zero

    var body: some View {
        Inner(outerOffset: $offsetY)
            .offset(y: offsetY)
            .gesture(DragGesture().onChanged { value in
                offsetY = value.translation.height
            }.onEnded { _ in
                offsetY = 0
            }
            )
    }

Inside the container we have our own ScrollView (ScrollViewOffset) which takes a callback with the current scroll offset. If the offset is positive (i.e. if we are scrolling upwards, even if we are at the top of the content), we want to disable the scroll and let the outer container drag along the y-axis instead of the ScrollView. To activate the ScrollView we listen for the outerOffset (the drag value) and activate the ScrollView when the offset is 0 again (this happens when releasing the drag as described before).

struct Inner: View {
    @Binding var outerOffset: CGFloat
    @State var disableScroll = false

    var body: some View {
        ScrollViewReader { proxy in
            ScrollViewOffset {
                ForEach(0 ... 10, id: \.self) { e in
                    Text(String(e))
                        .id(e)
                        .padding()
                    Divider()
                }
            } onOffsetChange: { offset in
                if offset > 0 {
                    disableScroll = true
                }
            }
            .background(.red)
            .padding()
            .frame(width: nil, height: 400)
            .onChange(of: outerOffset) { _ in
                if outerOffset == 0 {
                    proxy.scrollTo(0)
                    disableScroll = false
                }
            }
            .disabled(disableScroll)
        }
    }
}

When we run this, it is easy to see the problem I mentioned before. The ScrollView is getting disabled as it should, but the container is not focused on the drag. Only after starting a new drag, the container is focused.

My actual case

Last but not least the shazam equivalent (left) and my project (right).