Using geom_errorbar with color and size parameters for the same X value in ggplot2

Question

I am trying to create a scatterplot with categorical X values, where a X value is divided with color and shape parameter. The error bars seem to align with the color variable (due to the position_dodge) but this does not take the shape variable into account. What would the best way be to fix this issue? Create a seperate geom_errorbar for both shape variables? I am hoping that there might be an easier solution.

My data

Ind	Shape	Color	Mean	SE
A1	HQ	K1	0.034708	0.004165
A2	HQ	K1	0.084907	0.004624
A6	CQ	K2	0.00001	0
A3	HQ	K3	0.00001	0

My script

library(tidyverse)
library(ggplot2)

shapes <- c(23,22)

colors <- c("#ff9900","#3366cc","#dc3912")

check <- ggplot(data=data, aes(x=Ind, y=Mean)) +
  geom_point(aes(fill=Color, shape=Shape),colour='black',stat = "identity", position = position_dodge(0.9), size = 1.5, stroke=0.05, alpha=0.6) +
  geom_errorbar(aes(ymin = Mean - SE, ymax = Mean + SE, color = Color),stat = "identity", position = position_dodge(0.9), size = 0.21, stroke=0.05, width=0.31) +
  labs(x= '', y = "", size = 3) +
#  scale_y_continuous(limits=c(-0,1), expand=c(0,0)) +
  scale_fill_manual(values = colors) +
  scale_color_manual(values = colors) +
  scale_shape_manual(values = shapes ) + 
  theme_minimal() +
  theme(
    panel.grid = element_blank(),
    panel.background = element_rect(fill = NA, color = "black",size=0.25),
    axis.text.y = element_text(size=3.2),
    axis.text.x = element_text(size=3.2,angle = 90),
    strip.text.x = element_text(size = 3.2),
    legend.position = 'none',
    axis.ticks.y = element_line(size=0.2),
    axis.ticks.x = element_line(size=0.2)
  )

Also the plot. Scatterplot_errorbar

Thanks!