gg_vistime: How can I tweak the x axis without running into POSIXct problems?

Question

My x axis describes a timeline measured in years, with 50 years intervals. I want to set the starting year to 649 and the end year to 1258 and have them appear in the plot, and I try to use the scale_x_datetime function for this. I get an error message when I try to set the limits in that function.

Here is my code, built around the "tidslinje" dataframe:

                 gg_vistime(tidslinje,
                           linewidth = 5,
                           optimize_y = TRUE,
                           title = "Oversikt klassisk islamsk periode",
                           show_labels = TRUE,
                           background_lines = NULL) +
                           scale_x_datetime(date_breaks = "50 years",
                           date_labels = "%Y")

It produces this plot: gg_vistime plot, Islamic history timeline

This part of scale_x_datetime runs without problems:

scale_x_datetime(date_breaks = "50 years", date_labels = "%Y")

But the supposedly simple task of inserting a start year and an end year at either extremes of the x axis stumps me. Can I use scale_x_datetime for this? I have tried with the "limits" argument, but it tweaked the whole plot and didn't result in what I wanted.