How to match geom_ribbon fill to geom_line color (using ggplot2)?

Question

I am trying to create a line plot for two continuous variables (xy) with shading for the confidence interval, but with custom and matched colors for the line and the shading. GGplot correctly takes (discrete) colors from the viridis scale for the line, but I cannot get it to map the same colors (with lower alpha) to the geom_ribbon that I am using to plot the confidence interval.

Based on this post, I've tried using scale_color_manual but I can't get that to work either (even when I recode rej to character 'yes'/'no', or to a factor with those levels).

I've also tried setting the fill and color aesthetics directly in the call to ggplot, i.e.,

my_plot <- ggplot(data = my_data, aes(fill = rej, color = rej))

Here's the minimal example:

library(ggplot2)
library(viridis)

my_data <- structure(list(time = c(-0.1, 1.8, 3.7, 5.6, 7.5, 9.4, 11.3, 13.2, 
15.1, 17, -0.1, 1.8, 3.7, 5.6, 7.5, 9.4, 11.3, 13.2, 15.1, 17), value = c(-0.02, 
-0.05, -0.03, 0.03, 0.11, 0.11, 0.06, 0.01, -0.04, -0.18, -0.03, -0.05, -0.01, 
0.06, 0.14, 0.15, 0.09, 0.01, -0.07, -0.19), lower = c(-0.05, -0.12, -0.1, 
-0.05, 0.03, 0.04, -0.02, -0.07, -0.13, -0.33, -0.06, -0.12, -0.1, -0.03, 0.05, 
0.06, 0, -0.08, -0.16, -0.36), upper = c(0.02, 0.01, 0.05, 0.1, 0.18, 0.19, 
0.14, 0.1, 0.04, -0.02, 0.01, 0.03, 0.07, 0.14, 0.22, 0.23, 0.18, 0.1, 0.03, 
-0.01), rej = c(FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, 
FALSE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE)), 
row.names = c(11L, 12L, 13L, 14L, 15L, 16L, 17L, 18L, 19L, 20L, 41L, 42L, 43L, 
44L, 45L, 46L, 47L, 48L, 49L, 50L), class = "data.frame")

my_plot <- ggplot(data = my_data) +
  geom_ribbon(aes(x = time, ymin = lower, ymax = upper, fill = rej), alpha = 0.3) +
  geom_line(aes(x = time, y = value, color = rej), lwd = 3) + # line width increased for visibility
  scale_color_viridis_d(begin = 0, end = 1, option = 'viridis')
my_plot

Since the color argument for geom_line and the fill argument for geom_ribbon are set using the same variable, I am expecting the colors of lines and ribbons to match. What I get instead is a plot with lines and ribbons, but the colors of the ribbons do not match the colors of the lines (this can be confirmed by removing the alpha = 0.3 setting from the geom_ribbon arguments in the code above):

Is there any way to get ggplot to match the colors?