How to add significance bars between subgroups? Keeping the interaction

Question

I want to add significance bars between the subgroups of this plot, which has an interaction variable as a grouping variable. I would like to keep the interaction variable to stay consistent with my other plots.

This is my code so far

plot <- ggplot(data, aes(x = flow, y = var, group = interaction(sex,flow), fill = sex)) +
    stat_summary(fun.data=mean_se, geom="errorbar", color="black", width=0.3, ) +
    geom_quasirandom(data = . %>% filter(flow == "BF"),
                     size = 0.75, aes(color = sex), alpha = 0.6) +
    stat_summary(fun.y=mean, geom="crossbar",  color="black", size = 0.2, width = 0.5) +
    scale_color_manual(values = box_gradvec2) +
    scale_fill_manual(values = box_gradvec3) +
    ggtitle("Protein (FC)") +
    facet_wrap(~sex, strip.position = "bottom") + 
    expand_limits(y = 0) +
    theme_classic() +
    no_legend() + 
    theme(axis.title.x = element_blank(), axis.title.y = element_blank(), plot.title = element_text(face = "bold", hjust = 0.5), panel.spacing = unit(0, "lines"), strip.background = element_blank(), strip.placement = "outside", strip.text.x = element_text(margin = margin(b = -0.1)))

within_sex_p <- data %>% 
  group_by(sex) %>% 
  t_test(var ~ flow, paired = TRUE, alternative = "two.sided", p.adjust.method = "none") %>% 
  adjust_pvalue(method = "none") %>% 
  add_significance("p.adj") %>% 
  add_xy_position() %>% 
  filter(p < 0.05)

between_sex_p <- out_facs_mfi_fold_data %>% 
  group_by(flow) %>% 
  t_test(var ~ sex, paired = TRUE, alternative = "two.sided", p.adjust.method = "none") %>% 
  adjust_pvalue(method = "none") %>% 
  add_significance("p.adj") %>% 
  add_xy_position(x = "sex", dodge = 0.8, step.increase = 1.1) %>% 
  filter(p < 0.05)

plot <- plot + add_pvalue(within_sex_p, label = "p.adj.signif", inherit.aes = FALSE, tip.length = 0) + add_pvalue(between_sex_p, label = "p = {p}", inherit.aes = FALSE, tip.length = 0, xmin = "xmin", xmax = "xmax", step.group.by = "flow")

This code produces this plot:

I made another plot with the right placement of the significant bars, but since I want to keep the interaction variable to show, I cannot use it

There has been a question like before here, but there was no solution found, only workaround.