Do you need to Authorize everytime before calling any Gmail API?
I have done authorization once and got the access Token and Refresh Token.
If I call the API's every minute for a user, do I need to Refresh the tokens everytime? If not, when should I do it again?
The access token is valid for one hour then you can use the refresh token to get a new access token without having the user to grant permissions again.
you can use the refresh token whenever you want to get a new access token, although it would be better to use it just before the access token expires.
Here is more documentation on that.