Google Drive Rest API (V3) - 404 error while accessing private file owned by self

Question

I am trying to download (export) a Google Drive file owned by me (private - not shared with anyone) using release 21.3.0 of the nodejs client for Google API. I have set up a project in Google API Console and received a API key for the Drive API. When I try to access my file from a node program, I get a 'File not found' error. If I make the file public (On - Public on the web), I can access it fine. I have also tried setting up a service account to user server-to-server OAuth2 by following instructions here. I am then using the code in the examples to create a JWT token and use that to request a access token. But even with using that token, I get 'File not found' error when the file is private. Here is the code that I have so far for this functionality -

let key = require(#location_of_json_key_file);
let jwtClient = new google.auth.JWT(
    key.client_email,
    null,
    key.private_key,
    ['https://www.googleapis.com/auth/drive'],
    null
);

jwtClient.authorize(function (err, tokens) {
    if (err) {
        console.log(err);
        return;
    }

    console.log(tokens);
    let fileId = #id_of_file_on_google_drive; 
    googleDriveApi.files.export(
        {
            access_token: tokens.access_token, //tried auth: jwtClient as well
            fileId: fileId,
            mimeType: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
          //key: <API_access_key> //Tried with API Access Key
        },
        {encoding: null},
        (err, buffer) => {
            if (err) {
                //keep getting the error here - Error: File not found: #id_of_file_on_google_drive
                console.log(`Error occurred while exporting file from Google Drive - ${err}`);

            } else {
                //process the file...   
            }
        }
    );
});

If the file is owned by my account, should I not be able to access it using service account belonging to the same Google account even when the file is private? Otherwise, to make the Drive files accessible programmatically, you would have to make them public which is not always possible / advisable.

What am I missing here? Thanks for any pointers.