Read timed out. (read timeout=None) when inserting records through simple_salesforce API in Python

Question

I am getting a "ReadTimeout: HTTPSConnectionPool(host=' , port=443): Read timed out. (read timeout=None)" while inserting records using bulk api of simple_salesforce library of python. I have not set any timeout limit and I am trying to insert around 3M records. I also tried inserting in small chunks, but it has the same issue. Please let me know if any one of you have faced any such issues.

My Code:

from simple_salesforce import Salesforce, SalesforceLogin

sf = Salesforce(username=    ''    
                ,password=  ''                        
                ,security_token= '')


bulk_data = []
for row in <data_to_be_inserted>.itertuples():
  d = row._asdict()
  del d['Index']
  bulk_data.append(d)

sf.bulk.<Custom_Object_Name>.insert(bulk_data)

Answer 1

Yes, try increasing the timeout on the Requests session.

The simple_salesforce package uses Requests under the hood. Unfortunately, simple_salesforce does not appear to expose the timeout parameter.

Thankfully, it does allow you to customize the session and pass that into the Salesforce() constructor.

Here are two ways I found online to hack the session:

• Define a custom HTTP adapter
• This one seems easier: Use functools.partial

And a third and fourth:

• The Salesforce object has a "restful" method that can make arbitrary API calls. In other words, you can roll your own using, for instance, the Bulk API 2.0
• Once you are this far in, however, perhaps you want to make the API calls directly using the Requests library? It is a great library to work with, and you don't run into simple_salesforce's limitations.

I would love to be wrong about timeout. If anyone knows a way to set it from simple_salesforce, I am all ears.