How do I connect to a google sheet via googlesheets4 (r package) with a json oauth file?

Question

I need to connect to a google sheet in a non-interactive way. I've read the documentation here, here, and here and have done the following:

1. Created a google cloud account and project
2. Created a service account, and added the resulting email address as a editor to the sheet I'm trying to access (something I don't think any of the documentation even says you need to do, which is frustrating, because I'm pretty sure you have to do it)
3. Created an OAuth 2.0 Client ID
4. Downloaded the json file for that client ID

my code is:

library(googlesheets4)
gs4_auth_configure(path = "/path/to/my/service-account-token.json")
read_sheet("my_google_sheet_url.com")

but I still get asked to authenticate interactively, which is exactly what I'm trying to avoid. I've used the interactive mode, and then used gs4_auth(email = "my_email_address") which gets around that, but I need to run the script on a remote server where I can't do the initial interactive authentication required for that to work.