Combing through several large data structures of different classes in Python; How can I combine and store data I need while reducing memory usage?

Question

What's going on

I'm collecting data from a few thousand network devices every few minutes in Python 2.7.8 via package netsnmp. I'm also using fastsnmpy so that I can access the (more efficient) Net-SNMP command snmpbulkwalk.

I'm trying to cut down how much memory my script uses. I'm running three instances of the same script which sleeps for two minutes before re-querying all devices for data we want. When I created the original script in bash they would use less than 500MB when active simultaneously. As I've converted this over to Python, however, each instance hogs 4GB each which indicates (to me) that my data structures need to be managed more efficiently. Even when idle they're consuming a total of 4GB.

---

Code Activity

My script begins with creating a list where I open a file and append the hostname of our target devices as separate values. These usually contain 80 to 1200 names.

expand = []
f = open(self.deviceList, 'r')
for line in f:
    line = line.strip()
    expand.append(line)

From there I set up the SNMP sessions and execute the requests

expandsession = SnmpSession ( timeout = 1000000 ,
    retries = 1,            # I slightly modified the original fastsnmpy
    verbose = debug,        # to reduce verbose messages and limit
    oidlist = var,          # the number of attempts to reach devices
    targets = expand,
    community = 'expand'
)
expandresults = expandsession.multiwalk(mode = 'bulkwalk')

Because of how both SNMP packages behave, the device responses are parsed up into lists and stored into one giant data structure. For example,

for output in expandresults:
    print ouput.hostname, output.iid, output.val
#
host1 1 1
host1 2 2
host1 3 3
host2 1 4
host2 2 5
host2 3 6
# Object 'output' itself cannot be printed directly; the value returned from this is obscure
...

I'm having to iterate through each response, combine related data, then output each device's complete response. This is a bit difficult For example,

host1,1,2,3
host2,4,5,6
host3,7,8,9,10,11,12
host4,13,14
host5,15,16,17,18
...

Each device has a varying number of responses. I can't loop through expecting every device having a uniform arbitrary number of values to combine into a string to write out to a CSV.

---

How I'm handling the data

I believe it is here where I'm consuming a lot of memory but I cannot resolve how to simplify the process while simultaneously removing visited data.

expandarrays = dict()
for output in expandresults:
    if output.val is not None:
        if output.hostname in expandarrays:
            expandarrays[output.hostname] += ',' + output.val
        else:
            expandarrays[output.hostname] = ',' + output.val

for key in expandarrays:
    self.WriteOut(key,expandarrays[key])

Currently I'm creating a new dictionary, checking that the device response is not null, then appending the response value to a string that will be used to write out to the CSV file.

The problem with this is that I'm essentially cloning the existing dictionary, meaning I'm using twice as much system memory. I'd like to remove values that I've visited in expandresults when I move them to expandarrays so that I'm not using so much RAM. Is there an efficient method of doing this? Is there also a better way of reducing the complexity of my code so that it's easier to follow?

---

The Culprit

Thanks to those who answered. For those in the future that stumble across this thread due to experiencing similar issues: the fastsnmpy package is the culprit behind the large use of system memory. The multiwalk() function creates a thread for each host but does so all at once rather than putting some kind of upper limit. Since each instance of my script would handle up to 1200 devices that meant 1200 threads were instantiated and queued within just a few seconds. Using the bulkwalk() function was slower but still fast enough to suit my needs. The difference between the two was 4GB vs 250MB (of system memory use).

Answer 1

If the device responses are in order and are grouped together by host, then you don't need a dictionary, just three lists:

last_host = None
hosts = []                # the list of hosts
host_responses = []       # the list of responses for each host
responses = []
for output in expandresults:
    if output.val is not None:
        if output.hostname != last_host:    # new host
            if last_host:    # only append host_responses after a new host
                host_responses.append(responses)
            hosts.append(output.hostname)
            responses = [output.val]        # start the new list of responses
            last_host = output.hostname
        else:                               # same host, append the response
            responses.append(output.val)
host_responses.append(responses)

for host, responses in zip(hosts, host_responses):
    self.WriteOut(host, ','.join(responses))

Answer 2

The memory consumption was due to instantiation of several workers in an unbound manner.

I've updated fastsnmpy (latest is version 1.2.1 ) and uploaded it to PyPi. You can do a search from PyPi for 'fastsnmpy', or grab it directly from my PyPi page here at FastSNMPy

Just finished updating the docs, and posted them to the project page at fastSNMPy DOCS

What I basically did here is to replace the earlier model of unbound-workers with a process-pool from multiprocessing. This can be passed in as an argument, or defaults to 1.

You now have just 2 methods for simplicity. snmpwalk(processes=n) and snmpbulkwalk(processes=n)

You shouldn't see the memory issue anymore. If you do, please ping me on github.

Answer 3

You might have an easier time figuring out where the memory is going by using a profiler:

https://pypi.python.org/pypi/memory_profiler

Additionally, if you're already already tweaking the fastsnmpy classes, you can just change the implementation to do the dictionary based results merging for you instead of letting it construct a gigantic list first.

How long are you hanging on to the session? The result list will grow indefinitely if you reuse it.