Netbox ansible galaxy modules - inconsistent results

Question

I have just started to use netbox ansible modules in a vagrant ubuntu box. I am getting inconsistent results when I run the playbooks. Here is my configuration of the target box:

vagrant@netbox:~$ uname -a
Linux netbox 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Pynet version (from /sr/netbox/releases/netbox-3.5.3/netbox/netbox/settings.py):
VERSION = '3.5.3'

Here is the ansible script I am trying to run. Note: I have attempted to execute the script using python 3.8, 3.9, 3.10 and 3.11 with similar (inconsistent) behavior:

---
- hosts: netbox
  gather_facts: False
  connection: local

  vars:
    netbox_url: "http://192.168.56.5:80"
    netbox_token: "5929xxxx59"
    ansible_python_interpreter: /usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/bin/python3

    prefixes:
      - Prefix: 10.102.10.0/24
        Tenant: vnet-arc-fgt-dev-eu1

  tasks:
    - name: Create tenant within NetBox with only required information
      netbox.netbox.netbox_tenant:
        netbox_url: "{{ netbox_url }}"
        netbox_token: "{{ netbox_token }}"
        data:
          name: "{{ item.Tenant }}"
        state: present
        validate_certs: false
      with_items: "{{ prefixes }}"

    - name: Create prefix within NetBox with only required information
      netbox.netbox.netbox_prefix:
        netbox_url: "{{ netbox_url }}"
        netbox_token: "{{ netbox_token }}"

        data:
          prefix: "{{ item.Prefix }}"
          tenant: "{{ item.Tenant }}"
        state: present
      with_items: "{{ prefixes }}"

However I am getting inconsistent results: around 50% of the time, I get an error saying that the connection fails while at others the configuration is applied successfully:

successful response:

ansible-playbook -i inventory/vagrant-devtest.ini playbooks/ipam.yml

PLAY [netbox] ************************************************************************************************************************************************

TASK [Create tenant within NetBox with only required information] ********************************************************************************************
ok: [vagrant_netbox_server] => (item={'Prefix': '10.102.10.0/24', 'Tenant': 'vnet-arc-fgt-dev-eu1'})

TASK [Create prefix within NetBox with only required information] ********************************************************************************************
ok: [vagrant_netbox_server] => (item={'Prefix': '10.102.10.0/24', 'Tenant': 'vnet-arc-fgt-dev-eu1'})

PLAY RECAP ***************************************************************************************************************************************************
vagrant_netbox_server      : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Unsuccessful Response:

ansible-playbook -vvv -i inventory/vagrant-devtest.ini playbooks/ipam.yml 
ansible-playbook [core 2.15.1]
  config file = /Users/ypant/work/gcom/az-dev/devtest-ansible/ansible.cfg
  configured module search path = ['/Users/ypant/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/ansible
  ansible collection location = /Users/ypant/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/bin/ansible-playbook
  python version = 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:10:28) [Clang 15.0.7 ] (/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/bin/python)
  jinja version = 3.1.2
  libyaml = True
Using /Users/ypant/work/gcom/az-dev/devtest-ansible/ansible.cfg as config file
host_list declined parsing /Users/ypant/work/gcom/az-dev/devtest-ansible/inventory/vagrant-devtest.ini as it did not pass its verify_file() method
script declined parsing /Users/ypant/work/gcom/az-dev/devtest-ansible/inventory/vagrant-devtest.ini as it did not pass its verify_file() method
auto declined parsing /Users/ypant/work/gcom/az-dev/devtest-ansible/inventory/vagrant-devtest.ini as it did not pass its verify_file() method
yaml declined parsing /Users/ypant/work/gcom/az-dev/devtest-ansible/inventory/vagrant-devtest.ini as it did not pass its verify_file() method
Parsed /Users/ypant/work/gcom/az-dev/devtest-ansible/inventory/vagrant-devtest.ini inventory source with ini plugin
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: ipam.yml *************************************************************************************************************************************************************
1 plays in playbooks/ipam.yml

PLAY [netbox] ******************************************************************************************************************************************************************

TASK [Create tenant within NetBox with only required information] **************************************************************************************************************
task path: /Users/ypant/work/gcom/az-dev/devtest-ansible/playbooks/ipam.yml:16
<192.168.56.5> ESTABLISH LOCAL CONNECTION FOR USER: ypant
<192.168.56.5> EXEC /bin/sh -c 'echo ~ypant && sleep 0'
<192.168.56.5> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/ypant/.ansible/tmp `"&& mkdir "` echo /Users/ypant/.ansible/tmp/ansible-tmp-1689096350.351926-10450-86700633607771 `" && echo ansible-tmp-1689096350.351926-10450-86700633607771="` echo /Users/ypant/.ansible/tmp/ansible-tmp-1689096350.351926-10450-86700633607771 `" ) && sleep 0'
Using module file /Users/ypant/.ansible/collections/ansible_collections/netbox/netbox/plugins/modules/netbox_tenant.py
<192.168.56.5> PUT /Users/ypant/.ansible/tmp/ansible-local-10447mgwaldhj/tmpz9hdo77j TO /Users/ypant/.ansible/tmp/ansible-tmp-1689096350.351926-10450-86700633607771/AnsiballZ_netbox_tenant.py
<192.168.56.5> EXEC /bin/sh -c 'chmod u+x /Users/ypant/.ansible/tmp/ansible-tmp-1689096350.351926-10450-86700633607771/ /Users/ypant/.ansible/tmp/ansible-tmp-1689096350.351926-10450-86700633607771/AnsiballZ_netbox_tenant.py && sleep 0'
<192.168.56.5> EXEC /bin/sh -c '/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/bin/python3 /Users/ypant/.ansible/tmp/ansible-tmp-1689096350.351926-10450-86700633607771/AnsiballZ_netbox_tenant.py && sleep 0'
<192.168.56.5> EXEC /bin/sh -c 'rm -f -r /Users/ypant/.ansible/tmp/ansible-tmp-1689096350.351926-10450-86700633607771/ > /dev/null 2>&1 && sleep 0'
ok: [vagrant_netbox_server] => (item={'Prefix': '10.102.10.0/24', 'Tenant': 'vnet-arc-fgt-dev-eu1'}) => changed=false 
  ansible_loop_var: item
  invocation:
    module_args:
      cert: null
      data:
        comments: null
        custom_fields: null
        description: null
        name: vnet-arc-fgt-dev-eu1
        slug: null
        tags: null
        tenant_group: null
      netbox_token: VALUE_SPECIFIED_IN_NO_LOG_PARAMETER
      netbox_url: http://192.168.56.5:80
      query_params: null
      state: present
      validate_certs: false
  item:
    Prefix: 10.102.10.0/24
    Tenant: vnet-arc-fgt-dev-eu1
  msg: tenant vnet-arc-fgt-dev-eu1 already exists
  tenant:
    circuit_count: 0
    cluster_count: 0
    comments: ''
    created: '2023-07-10T14:49:19.470354Z'
    custom_fields: {}
    description: ''
    device_count: 0
    display: vnet-arc-fgt-dev-eu1
    group: null
    id: 1
    ipaddress_count: 0
    last_updated: '2023-07-10T14:49:19.470380Z'
    name: vnet-arc-fgt-dev-eu1
    prefix_count: 1
    rack_count: 0
    site_count: 0
    slug: vnet-arc-fgt-dev-eu1
    tags: []
    url: http://192.168.56.5/api/tenancy/tenants/1/
    virtualmachine_count: 0
    vlan_count: 0
    vrf_count: 0

TASK [Create prefix within NetBox with only required information] **************************************************************************************************************
task path: /Users/ypant/work/gcom/az-dev/devtest-ansible/playbooks/ipam.yml:26
<192.168.56.5> ESTABLISH LOCAL CONNECTION FOR USER: ypant
<192.168.56.5> EXEC /bin/sh -c 'echo ~ypant && sleep 0'
<192.168.56.5> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/ypant/.ansible/tmp `"&& mkdir "` echo /Users/ypant/.ansible/tmp/ansible-tmp-1689096351.386289-10470-259184315756197 `" && echo ansible-tmp-1689096351.386289-10470-259184315756197="` echo /Users/ypant/.ansible/tmp/ansible-tmp-1689096351.386289-10470-259184315756197 `" ) && sleep 0'
Using module file /Users/ypant/.ansible/collections/ansible_collections/netbox/netbox/plugins/modules/netbox_prefix.py
<192.168.56.5> PUT /Users/ypant/.ansible/tmp/ansible-local-10447mgwaldhj/tmpxkjnxavx TO /Users/ypant/.ansible/tmp/ansible-tmp-1689096351.386289-10470-259184315756197/AnsiballZ_netbox_prefix.py
<192.168.56.5> EXEC /bin/sh -c 'chmod u+x /Users/ypant/.ansible/tmp/ansible-tmp-1689096351.386289-10470-259184315756197/ /Users/ypant/.ansible/tmp/ansible-tmp-1689096351.386289-10470-259184315756197/AnsiballZ_netbox_prefix.py && sleep 0'
<192.168.56.5> EXEC /bin/sh -c '/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/bin/python3 /Users/ypant/.ansible/tmp/ansible-tmp-1689096351.386289-10470-259184315756197/AnsiballZ_netbox_prefix.py && sleep 0'
<192.168.56.5> EXEC /bin/sh -c 'rm -f -r /Users/ypant/.ansible/tmp/ansible-tmp-1689096351.386289-10470-259184315756197/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/connection.py", line 454, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/http/client.py", line 1378, in getresponse
    response.begin()
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/connection.py", line 454, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/http/client.py", line 1378, in getresponse
    response.begin()
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ypant/.ansible/tmp/ansible-tmp-1689096351.386289-10470-259184315756197/AnsiballZ_netbox_prefix.py", line 107, in <module>
    _ansiballz_main()
  File "/Users/ypant/.ansible/tmp/ansible-tmp-1689096351.386289-10470-259184315756197/AnsiballZ_netbox_prefix.py", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/Users/ypant/.ansible/tmp/ansible-tmp-1689096351.386289-10470-259184315756197/AnsiballZ_netbox_prefix.py", line 47, in invoke_module
    runpy.run_module(mod_name='ansible_collections.netbox.netbox.plugins.modules.netbox_prefix', init_globals=dict(_module_fqn='ansible_collections.netbox.netbox.plugins.modules.netbox_prefix', _modlib_path=modlib_path),
  File "<frozen runpy>", line 226, in run_module
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/var/folders/k8/cnw1b0ls2f90q8dwkyz720200000gn/T/ansible_netbox.netbox.netbox_prefix_payload_hdayngnd/ansible_netbox.netbox.netbox_prefix_payload.zip/ansible_collections/netbox/netbox/plugins/modules/netbox_prefix.py", line 288, in <module>
  File "/var/folders/k8/cnw1b0ls2f90q8dwkyz720200000gn/T/ansible_netbox.netbox.netbox_prefix_payload_hdayngnd/ansible_netbox.netbox.netbox_prefix_payload.zip/ansible_collections/netbox/netbox/plugins/modules/netbox_prefix.py", line 283, in main
  File "/var/folders/k8/cnw1b0ls2f90q8dwkyz720200000gn/T/ansible_netbox.netbox.netbox_prefix_payload_hdayngnd/ansible_netbox.netbox.netbox_prefix_payload.zip/ansible_collections/netbox/netbox/plugins/module_utils/netbox_ipam.py", line 41, in __init__
  File "/var/folders/k8/cnw1b0ls2f90q8dwkyz720200000gn/T/ansible_netbox.netbox.netbox_prefix_payload_hdayngnd/ansible_netbox.netbox.netbox_prefix_payload.zip/ansible_collections/netbox/netbox/plugins/module_utils/netbox_utils.py", line 689, in __init__
  File "/var/folders/k8/cnw1b0ls2f90q8dwkyz720200000gn/T/ansible_netbox.netbox.netbox_prefix_payload_hdayngnd/ansible_netbox.netbox.netbox_prefix_payload.zip/ansible_collections/netbox/netbox/plugins/module_utils/netbox_utils.py", line 1203, in _find_ids
  File "/var/folders/k8/cnw1b0ls2f90q8dwkyz720200000gn/T/ansible_netbox.netbox.netbox_prefix_payload_hdayngnd/ansible_netbox.netbox.netbox_prefix_payload.zip/ansible_collections/netbox/netbox/plugins/module_utils/netbox_utils.py", line 746, in _nb_endpoint_get
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/pynetbox/core/endpoint.py", line 161, in get
    ret = next(resp, None)
          ^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/pynetbox/core/response.py", line 127, in __next__
    next(self.response), self.endpoint.api, self.endpoint
    ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/pynetbox/core/query.py", line 280, in get
    req = self._make_call(add_params=add_params)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/pynetbox/core/query.py", line 230, in _make_call
    req = getattr(self.http_session, verb)(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
failed: [vagrant_netbox_server] (item={'Prefix': '10.102.10.0/24', 'Tenant': 'vnet-arc-fgt-dev-eu1'}) => changed=false 
  ansible_loop_var: item
  item:
    Prefix: 10.102.10.0/24
    Tenant: vnet-arc-fgt-dev-eu1
  module_stderr: |-
    Traceback (most recent call last):
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
        response = self._make_request(
                   ^^^^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
        response = conn.getresponse()
                   ^^^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/connection.py", line 454, in getresponse
        httplib_response = super().getresponse()
                           ^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/http/client.py", line 1378, in getresponse
        response.begin()
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/http/client.py", line 318, in begin
        version, status, reason = self._read_status()
                                  ^^^^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/http/client.py", line 287, in _read_status
        raise RemoteDisconnected("Remote end closed connection without"
    http.client.RemoteDisconnected: Remote end closed connection without response
  
    During handling of the above exception, another exception occurred:
  
    Traceback (most recent call last):
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
        resp = conn.urlopen(
               ^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/connectionpool.py", line 844, in urlopen
        retries = retries.increment(
                  ^^^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
        raise reraise(type(error), error, _stacktrace)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
        raise value.with_traceback(tb)
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
        response = self._make_request(
                   ^^^^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
        response = conn.getresponse()
                   ^^^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/urllib3/connection.py", line 454, in getresponse
        httplib_response = super().getresponse()
                           ^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/http/client.py", line 1378, in getresponse
        response.begin()
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/http/client.py", line 318, in begin
        version, status, reason = self._read_status()
                                  ^^^^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/http/client.py", line 287, in _read_status
        raise RemoteDisconnected("Remote end closed connection without"
    urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  
    During handling of the above exception, another exception occurred:
  
    Traceback (most recent call last):
      File "/Users/ypant/.ansible/tmp/ansible-tmp-1689096351.386289-10470-259184315756197/AnsiballZ_netbox_prefix.py", line 107, in <module>
        _ansiballz_main()
      File "/Users/ypant/.ansible/tmp/ansible-tmp-1689096351.386289-10470-259184315756197/AnsiballZ_netbox_prefix.py", line 99, in _ansiballz_main
        invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
      File "/Users/ypant/.ansible/tmp/ansible-tmp-1689096351.386289-10470-259184315756197/AnsiballZ_netbox_prefix.py", line 47, in invoke_module
        runpy.run_module(mod_name='ansible_collections.netbox.netbox.plugins.modules.netbox_prefix', init_globals=dict(_module_fqn='ansible_collections.netbox.netbox.plugins.modules.netbox_prefix', _modlib_path=modlib_path),
      File "<frozen runpy>", line 226, in run_module
      File "<frozen runpy>", line 98, in _run_module_code
      File "<frozen runpy>", line 88, in _run_code
      File "/var/folders/k8/cnw1b0ls2f90q8dwkyz720200000gn/T/ansible_netbox.netbox.netbox_prefix_payload_hdayngnd/ansible_netbox.netbox.netbox_prefix_payload.zip/ansible_collections/netbox/netbox/plugins/modules/netbox_prefix.py", line 288, in <module>
      File "/var/folders/k8/cnw1b0ls2f90q8dwkyz720200000gn/T/ansible_netbox.netbox.netbox_prefix_payload_hdayngnd/ansible_netbox.netbox.netbox_prefix_payload.zip/ansible_collections/netbox/netbox/plugins/modules/netbox_prefix.py", line 283, in main
      File "/var/folders/k8/cnw1b0ls2f90q8dwkyz720200000gn/T/ansible_netbox.netbox.netbox_prefix_payload_hdayngnd/ansible_netbox.netbox.netbox_prefix_payload.zip/ansible_collections/netbox/netbox/plugins/module_utils/netbox_ipam.py", line 41, in __init__
      File "/var/folders/k8/cnw1b0ls2f90q8dwkyz720200000gn/T/ansible_netbox.netbox.netbox_prefix_payload_hdayngnd/ansible_netbox.netbox.netbox_prefix_payload.zip/ansible_collections/netbox/netbox/plugins/module_utils/netbox_utils.py", line 689, in __init__
      File "/var/folders/k8/cnw1b0ls2f90q8dwkyz720200000gn/T/ansible_netbox.netbox.netbox_prefix_payload_hdayngnd/ansible_netbox.netbox.netbox_prefix_payload.zip/ansible_collections/netbox/netbox/plugins/module_utils/netbox_utils.py", line 1203, in _find_ids
      File "/var/folders/k8/cnw1b0ls2f90q8dwkyz720200000gn/T/ansible_netbox.netbox.netbox_prefix_payload_hdayngnd/ansible_netbox.netbox.netbox_prefix_payload.zip/ansible_collections/netbox/netbox/plugins/module_utils/netbox_utils.py", line 746, in _nb_endpoint_get
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/pynetbox/core/endpoint.py", line 161, in get
        ret = next(resp, None)
              ^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/pynetbox/core/response.py", line 127, in __next__
        next(self.response), self.endpoint.api, self.endpoint
        ^^^^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/pynetbox/core/query.py", line 280, in get
        req = self._make_call(add_params=add_params)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/pynetbox/core/query.py", line 230, in _make_call
        req = getattr(self.http_session, verb)(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
        return self.request("GET", url, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
        resp = self.send(prep, **send_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
        r = adapter.send(request, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/Caskroom/miniconda/base/envs/ansible_py3.10/lib/python3.11/site-packages/requests/adapters.py", line 501, in send
        raise ConnectionError(err, request=request)
    requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  module_stdout: ''
  msg: |-
    MODULE FAILURE
    See stdout/stderr for the exact error
  rc: 1

PLAY RECAP *********************************************************************************************************************************************************************
vagrant_netbox_server      : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

Update: Jul 11, 2023 @ 6:00pm America/Chicago Time

I captured packets between ansible controller (192.168.56.3) and the netbox server (192.168.56.5).

Ansible controller (.3 server) makes http call (packet 55) makes an api call to get /api/tenancy/tenants, however the netbox server (.5) simply resets the connection.

However there is no log entry corresponding to the packet error above:

2023-07-11 22:05:12,421 netbox.rqworker WARNING: No queues have been specified. This process will service the following queues by default: high, default, low
2023-07-11 22:13:09,095 django.request WARNING: Not Found: /asdaffa
2023-07-11 22:18:51,628 netbox.rqworker WARNING: No queues have been specified. This process will service the following queues by default: high, default, low
2023-07-11 22:25:43,867 django.request WARNING: Not Found: /ipam/ip-ranges/2/
2023-07-11 22:27:53,344 django.request WARNING: Not Found: /ipam/ip-ranges10

Answer 1

I made several attempts to resolve the issue, including the following steps:

• Resizing the Netbox VM.
• Introducing arbitrary delays in the playbooks
• Experimenting with different operating systems such as Ubuntu 20.04, 22.04, CentOS, Rocky Linux, etc.

Unfortunately, despite trying various approaches, the connection reset issue remained unresolved. Ideally, using an Ansible playbook would have been a better solution. However, as a temporary workaround, I created a Python script that introduces delays between API calls. I'm sharing this code because I invested significant time and effort into finding a resolution, and I believe it could be helpful for others encountering similar challenges in their work. In a couple of cases, I had to run the script multiple times to successfully provision netbox.

You can also find the code on GitHub at the following link: https://github.com/ypant/netbox/blob/main/netbox_import_csv.py

import pynetbox
import csv
import time
from pprint import pprint
import re

SLEEP_TIME=3    # time to pause between each api call  
IP_NETBOX_FILE = './ip-netbox.csv'

'''
# example format of the csv file, comments start in #

# column indices 0,1,2,3,4,5,6,7,8,9
#,Aggregate,Prefix,Tenant,TenantGroup,Status,Site,Description,Tags,Comments
# Example,10.103.0.0/16,10.103.10.0/23,"In the new devtest environment, IP allocated for ….","choose one of devtest, prod, staging, qa","Active, container, Reserved, Deprecated","aws, azure, onprem","Resource group (azure) or account id (aws) prefixed by ""a-""",,
,,10.102.10.0/24,vnet-dev,devtest,active,azure,rg-dev,eastus,
,,10.102.20.0/24,vnet-prod,prod,active,azure,rg-prod,eastus,
,,10.102.0.0/24,vnet-staging,staging,active,azure,rg-staging,eastus,
,,10.0.0.0/16,vnet-test,qa,active,azure,rg-qa,eastus,

'''

netbox_url="http://198.168.56.76:80"
netbox_token="263136752a7d0a95d6xxxxxx4d828419b0b014"

def main(argv=None):
  update_tenant_groups()
  update_tenants()
  update_prefixes()

# function for creating slug names consistentnly 
def get_slug(name):
  name = name.strip()

  char_to_remove = ["-", " "]
  for char in char_to_remove:
    name = name.replace(char, "_")

  name = name.lower()
  name = re.sub('_+', '_', name)
  return name

def update_tenant_groups():
  print("Inside tenant_groups")
  nbapi = pynetbox.api(url=netbox_url, token=netbox_token) 

  #print("Getting all tenant groups")
  tgs = nbapi.tenancy.tenant_groups.all()

  # create an empty hash
  current_tgs = {}
  for tg in tgs:
    tg_slug = get_slug(str(tg))
    current_tgs[tg_slug] = ""
  #  print(str(tg))

  with open(IP_NETBOX_FILE) as csv_file:
    csv_reader = csv.reader(csv_file, delimiter=',')
    #print(row)
    for row in csv_reader:
      if ( '#' in row[0] ):
        #print ("Skipping " + str(row))
        continue
      else:
        #print ("Processing " + str (row))

        tg = get_slug(row[4].strip())
        #print ("tg " + tg)

        # Create tenant group only if non-empty
        if (tg != ''):
          if tg in current_tgs:
            print ("Tenant Group [" + tg + "] already exists ")
          else:
            print ("Creating tenant group [" + tg + "] ")
            #tg_slug = get_slug(tg)
            print("Adding: [" + tg + "]")
            create_tg = nbapi.tenancy.tenant_groups.create(name=tg, slug=tg)
            time.sleep(SLEEP_TIME) 

            print("Added: [" + str(create_tg) + " " + tg + "]")
            current_tgs[str(tg)] = ""

  print ("Imported tenant group from CSV File")

def update_tenants():
  print("Inside tenants")
  time.sleep(SLEEP_TIME) 
  nbapi = pynetbox.api(url=netbox_url, token=netbox_token) 

  #print("Getting all tenants")
  tenants = nbapi.tenancy.tenants.all()

  # create an empty hash
  current_tenants = {}
  for tenant in tenants:
    tenant_slug=get_slug(str(tenant))
    current_tenants[tenant_slug] = ""
  #  print(str(tg))

  with open(IP_NETBOX_FILE) as csv_file:
    csv_reader = csv.reader(csv_file, delimiter=',')
    #print(row)
    for row in csv_reader:
      if ( '#' in row[0] ):
        #print ("Skipping " + str(row))
        continue
      else:
        #print ("Processing " + str (row))

        tenant_slug = get_slug(row[3].strip())
        #print ("tg " + tg)

        # Create tenant group only if non-empty
        if (tenant_slug != ''):
          if tenant_slug in current_tenants:
            print ("Tenant [" + tenant_slug + "] already exists ")
          else:
            group_slug=get_slug(row[4].strip())
            print ("Creating tenant [" + tenant_slug + "] ")

            time.sleep(SLEEP_TIME) 
            tenant_group = nbapi.tenancy.tenant_groups.get(slug=group_slug)
            #print ("Slug here = " + tenant)
            #print (tenant_id.id)
            print ("retrieved tenant group id = " + str(tenant_group.id) )

            time.sleep(SLEEP_TIME) 
            try:
              create_tenant = nbapi.tenancy.tenants.create(name=tenant_slug, slug=tenant_slug, group=tenant_group.id)
            except Exception as e:
              print("Exception occurred...")
              print(type(e))
              print(e)
              time.sleep(2*SLEEP_TIME)
              continue

            print("Added: [" + str(create_tenant) + "]")
            current_tenants[tenant_slug] = ""

  print ("Imported tenants from CSV File")

def update_prefixes():
  print("Inside update_prefixes ")
  nbapi = pynetbox.api(url=netbox_url, token=netbox_token) 

  #print("Getting all prefixes")
  prefixes = nbapi.ipam.prefixes.all()

  # create an empty hash
  current_prefixes = {}
  for p in prefixes:
    tenant_slug = get_slug(str(p.tenant))
    prefix_slug = get_slug(str(p))
    current_prefixes[prefix_slug] = tenant_slug
    print('Adding hash: ' + prefix_slug + ' with value ' + tenant_slug )

  with open(IP_NETBOX_FILE) as csv_file:
    csv_reader = csv.reader(csv_file, delimiter=',')
    #print(row)
    for row in csv_reader:
      if ( '#' in row[0] ):
        #print ("Skipping " + str(row))
        continue
      else:
        #print ("Processing " + str (row))

        prefix = get_slug(row[2])
        tenant = get_slug(row[3])

        print ("Inside else: " + prefix + ' ' + tenant)

        #print ("tg " + tg)

        # Create tenant group only if non-empty
        if (prefix != ''):
          #print (prefix + ' value ' + current_prefixes[prefix] + ' ' + tenant)

          if ((prefix in current_prefixes) and (tenant == current_prefixes[prefix])):
            print ("Prefix [" + prefix + "] with value [" + str(current_prefixes[str(prefix)]) + " already exists in [tenant]... skipping...")
          else:
            print ("Creating prefixes [" + prefix + "], tenant = " + tenant)

            tenant_id = nbapi.tenancy.tenants.get(slug=tenant)
            print ("Tenant slug here = " + tenant)
            print ("retrieved tenant id = " + str(tenant_id.id) )

            time.sleep(SLEEP_TIME) 
            create_prefix = nbapi.ipam.prefixes.create(prefix=prefix, tenant=tenant_id.id, 
              status=row[5], #site=row[6],tags=row[8],
              description=row[7], comments=row[9])
            time.sleep(SLEEP_TIME) 
            print("Added: [" + str(create_prefix) + "]")
            current_prefixes[prefix] = tenant

  print ("Imported ip prefixes from CSV File") 
 
if __name__ == "__main__":
  main()