Google APIs Refresh Token Limitations

Question

I have developed an application to scan user's drive on behalf of the user. After refreshing token for few times I got {'invalid_grant':'Token has been expired or revoke.'}. I confirmed the refresh token's limitations from the user but he did no change. What could be happened and what is the solution for this issue. Kindly guide me.

I have tried refreshing the token using Google Drive's Python scrip:

from google.oauth2.credentials import Credentials

creds = Credentials(
    credentials["access_token"],
    refresh_token=credentials["refresh_token"],
    token_uri=GDRIVE_TOKEN_URI,
    client_id=GDRIVE_CLIENT_ID,
    client_secret=GDRIVE_CLIENT_SECRET,
    scopes=GOOGLEDRIVE_SCOPES,
)
creds.refresh(Request())
cred = creds.to_json()

Every time I refreshes the token I save latest access token and refresh token in my database.