FAILD status when export query result from Google Vault API

Question

I’m using Google Vault API (python script) to query message data from google chat. I have about 1800 emails but when I queried from the 1000th email to 1050th email, the export status returned FAILED without any error message and there was no error message. Then when I queried from 1050th to 1100th email, the file was exported normally.

Considering that there are some anomalies in the emails from 1000 to 1050, I tried to queried from 1000 to 1010, from 1010 to 1020,… from 1040 to 1050. But at this time they all worked!

I need to build a system to automate this process, so I cannot do it manually like this and query by each 10 emails is so time-consuming. So please help me to figure out what was wrong so that I can query all 1800 emails at once without FAIED status.

Here is my query code:

def create_mail_all_data_export(service, matter_id,start_date,end_date,name):
    df = pd.read_csv("data_users_2.csv")
    email_list = list(df["email"])
    email_str = ",".join(email_list)
    #print(email_str)
    """
    全社員のメールエクスポートを作成
    Args:
        service : v1 vaultでbuildされたサービス
        matter_id : vault案件id
        start_date : 検索開始日時
        end_date : 検索修了日時
        name : ファイル名接頭語
    """

    hangout_query_options = {"includeRooms": True}
    hangout_query = {
        'corpus': 'HANGOUTS_CHAT',
        'dataScope': 'ALL_DATA',
        'searchMethod': 'ACCOUNT',
        'accountInfo': {
            "emails": [email_list[1000:1050]]
            # "emails": [
            #     "[email protected]",
            #     "[email protected]"
            #     #email_list
            # ]
        },
        'timeZone':"Japan",
        "startTime":start_date,
        "endTime":end_date,
        'hangoutsChatOptions': hangout_query_options,
    }
    hangout_export_options = {
      'exportFormat': 'MBOX'
      }
    wanted_export = {
        'name': name,
        'query': hangout_query,
        'exportOptions': {
            'hangoutsChatOptions': hangout_export_options
            }
        }
    return service.matters().exports().create(
    matterId=matter_id, body=wanted_export).execute()

Answer 1

Most likely you running into quota limit issues

Check your quota and compare them with your usage.

For this

• Go to your Cloud Console

• Find the G Suite Vault API under https://console.cloud.google.com/apis/api/vault.googleapis.com/overview?project=**NAME_OF_YOUR_PROJECT**

• Check your metrics:

---

• And your quota:

---

• If you expand the different categories, you can also see your current quota per minute:

• You can try to edit the quote by clicking on the "pencil"

• Or Request more quota limits or view quotas for your other services on the Quotas page, found in IAM & admin. under https://console.cloud.google.com/iam-admin/quotas?service=vault.googleapis.com&project=**NAME_OF_YOUR_PROJECT**

However, if you are exceeding the per minute quota, the best solution would be to modify your code in such a way that it runs slower. A good way to do so would be to use Exponential Backoff.

Note: The per minute quotas actually can be exceeded for a short amount of time, but once they are exceeded they can take up to 24 hours to be established.