react-native-gesture-handler setState very slow inside gesture handler

Question

I'm in the process of upgrading from react-native-gesture-handler version 1 to version 2.

In my gesture handler, I set state variables, which is causing a substantial slow down. This was not occurring in the version 1 handler.

Here the basics:

    const panGesture = Gesture.Pan()
        .runOnJS(true)
        .onUpdate((e) => {
            console.log('over scroll', e.translationY)
            const offsetY = e.translationY
            if (scrollIsTop && offsetY > 0) {
                setHeaderMarginTop(initialHeaderMarginTop + offsetY )
                setContentMarginTop(initialContentMarginTop + offsetY )
            }
        })
        .simultaneousWithExternalGesture(scrollRef)

    return (
        <SwipeUpDownModal
            modalVisible={visible}
            PressToanimate={animateModal}
            HeaderStyle={{marginTop:headerMarginTop}}
            ContentModalStyle={{marginTop: contentMarginTop}}
            ContentModal={              
                    <GestureDetector gesture={panGesture}>
                        <ScrollView
                            ref={scrollRef}
                            style={styles.modalContent}
                            scrollEventThrottle={16}
                            onScroll={checkOverScroll}
                            bounces={false}
                        >
                         ...
                        </ScrollView>
                     </GestureHandler>
        </SwipeUpDownModal>
    )

The setHeaderMarginTop() and setContentMarginTop() makes the process crawl. If I take out the state changes, then then everything runs fast.

Any ideas how to get around this slow down?