How to improve read remote system files with high performance in python

Question

I have written code for reading the remote system files that is working fine but it is taking much time to read remote files. I want to improve performance for reading.

I have also used python Threading for reading the remote system files. that is also taking much time. now anyone let me know better suggestion for that.

I have used this code for reading remote system file,

    root_folder="\\\\192.168.1.1\\C$"
    try:
        use_dict={}
        use_dict['remote']=unicode(root_folder)
        use_dict['password']=unicode("example")
        use_dict['username']=unicode("example")
        win32net.NetUseAdd(None, 2, use_dict)
        print "Network connection established"
    except:
        print "Network connection failed"

    for root, dirnames, filenames in os.walk(root_folder):
       for filename in filenames:
           match=os.path.join(root, filename)
           datafile = file(match)
           for line in datafile:
              for li in line:
                  print li

from this code, 45 min time is taking for reading the remote system files. and If i read same files in local way then it is taking only 5 min. so I am not able to increase performance to read. please let me know for increase performance for reading...

thanks...

Answer 1

You can try multiprocessing. In this example a process reads from network and the other one prints data and they are connected by a Queue.

from multiprocessing import Process, Queue

def readfiles(q):

    root_folder="\\\\192.168.1.1\\C$"
    try:
        use_dict={}
        use_dict['remote']=unicode(root_folder)
        use_dict['password']=unicode("example")
        use_dict['username']=unicode("example")
        win32net.NetUseAdd(None, 2, use_dict)
        print "Network connection established"
    except:
        print "Network connection failed"

    for root, dirnames, filenames in os.walk(root_folder):
        for filename in filenames:
            match=os.path.join(root, filename)
            datafile = file(match)
            for line in datafile:
                q.put(line)
    q.close()


if __name__ == '__main__':
    q = Queue()
    p = Process(target=readfiles, args=(q,))
    p.start()

    while p.is_alive() or not q.empty():
        for li in q.get():
            print li