List update with swipable_stack / Flutter - Riverpod

Question

I'm working on an app with a swipe feature. I'm using swipable_stack and Riverpod as state management.

My app is build like there are 2 section : Swipe and Grid who share the same list (to let user uses which mode he likes). I get the users list in Firebase and this list is shared in both section. To have a good UX, when user swipes someone, the user who was swiped needs to be remove from the users list, so he does not appear in the other section.

My problem here is when I remove a user from the list after a swipe. I have a micro-second where I can see the user behind the new one displayed. And when I remove the user deletion from my code, it works fine, but the user is always visible on the other section.

Here is my code :

SwipableStack(
                  swipeAnchor: SwipeAnchor.bottom,
                  itemCount: otherUserList.length,
                  detectableSwipeDirections: const {
                    SwipeDirection.right,
                    SwipeDirection.left,
                  },
                  controller: _controller,
                  stackClipBehaviour: Clip.antiAlias,
                  allowVerticalSwipe: false,
                  onSwipeCompleted: (index, direction) async {
                    final otherUser = otherUserList[index];
                    final otherUserId = otherUser.uid;
                    print("${otherUser.pseudo} swiped !");

                    if (direction == SwipeDirection.right) {
                      
                      if (otherUserId != null) {
                        addToLikeList(currentUserAuth.uid, otherUserId);
                        final List<OtherUserModel> updatedUserList =
                            List.from(otherUserList);
                        updatedUserList.remove(otherUser);
                        ref.read(otherUserListProvider.notifier).state =
                            updatedUserList;
                        checkMatch(otherUser, ref);
                      }
                    }

                    if (direction == SwipeDirection.left) {
                      if (otherUserId != null) {
                        addToDislikeList(currentUserAuth.uid, otherUserId);
                        final List<OtherUserModel> updatedUserList =
                            List.from(otherUserList);
                        updatedUserList.remove(otherUser);
                        ref.read(otherUserListProvider.notifier).state =
                            updatedUserList;
                      }
                    }
                  },
                  builder: (context, properties) {
                    if (properties.index >= otherUserList.length) {
                      return const SizedBox.shrink();
                    }

                    final itemIndex = properties.index;
                    return Stack(
                      children: [
                        Padding(
                          padding: const EdgeInsets.all(10.0),
                          child: SwipeCard(
                            uid: otherUserList[itemIndex].uid!,
                          ),
                        ),
                        if (properties.stackIndex == 0 &&
                            properties.direction != null)
                          CardOverlay(
                            swipeProgress: properties.swipeProgress,
                            direction: properties.direction!,
                          ),
                      ],
                    );
                  },
                ),

I am not using Riverpod for a long time so there may be some mistakes. I can provide more code of information if needed.