How can I combine two box plots with different dataframes into one graph with secondary axis in R?

Question

##Individual plots

par(mfrow = c(1,2))  

dat <- data.frame(years = rep(2011:2020, each = 9), x = sample(200:325, 10*9 ,replace = T))  
boxplot(dat$x ~ dat$year, ylim = c(200, 350)) 

#reference data  
ref.dat <- data.frame(years = rep(2011:2020), x = sample(0:40, 10, replace = T))  
plot(ref.dat$years, ref.dat$x, type = "b")

ylim.prim <- c(200, 325)     
ylim.sec <- c(0, 40)      

b <- diff(ylim.prim)/diff(ylim.sec)    
a <- ylim.prim[1] - b*ylim.sec[1]     

ggplot(dat, aes(x = years, y = x)) +
  geom_boxplot(data = dat,  aes(group = years)) +
  geom_line(data = ref.dat, aes(x = years, y = X), color = "red") +
              scale_y_continuous("Sample", sec.axis = sec_axis(~ (. - a)/b, name = "reference")) 

Now how can i overlay this line over the box plot with a secondary axis.