Change font size for entire gt() table in quarto with Latex / PDF output?

Question

I am creating a document with Quarto and want to change the font size of my tables made with the gt() package.
While this works fine for HTML output, it has no effect for PDF output.
Does anyone know a solution?

---
format: pdf
---

```{r}
library(gt)
gt(mtcars) |> tab_options(table.font.size = 7)
```
```{r}
gt(mtcars) |> tab_options(table.font.size = 12)
```