Gesture with modifier key on a ColorPicker

Question

In my macOS app, I have a view with a ColorPicker and I am trying to install a gesture with modifier key, like this:

struct ContentView: View {
    @State private var backgroundColor: Color = .gray
    
    var body: some View {
        VStack {
            ColorPicker("Choose background color", selection: $backgroundColor, supportsOpacity: false)
                .gesture(TapGesture().modifiers(.option).onEnded {
                    print("ColorPicker was Option-clicked")
                    backgroundColor = .gray
                })
        }
        .padding()
        .fixedSize()
        .background(backgroundColor)
    }
}

I would like to have this option-click work on the whole ColorPicker view, however it only works on the label, not on the color well.

I tried to use highPriorityGesture instead of gesture but the result is the same.