Need tips to improve performance of Python program

Question

I am creating a simulator tool in python which will send radius request using pyrad module.

But I am facing some performance issue, where the performance is only 1500 tps. Please share some ideas/suggestions to scale up the performance of simulator.

class MyThread(threading.Thread):

    def __init__(self, auth_obj):
        threading.Thread.__init__(self)
        self.auth_obj = auth_obj

    def run(self):
        self.auth_obj.create_threads()


class Authenticator:
    def __init__(self):
        self.iterations = 0
        self.concurrency = 0
        self.secret_pwd = ""
        self.server_ip = ""

        self.read_xml()

        self.user_idx_list = range(self.concurrency)
        if self.concurrency > MAX_CONC_THREADS:
            self.concurrency = MAX_CONC_THREADS

        print("Main:Starting Time: {}".format((datetime.now()).strftime("%B %d, %H:%M:%S %Y")))

        # Using Threads
        thread_list = list()

        for index in range(self.iterations):
            thread = MyThread(self)
            thread_list.append(thread)
            thread.start()
            time.sleep(1)

            if index % PRINT_OUTPUT_SEC == 0:
                print_table_output()

        for cur_t in thread_list:
            cur_t.join()

        print_table_output()
        str_out = " Final Stats "
        print(str_out.center(63, '#'))
        input("Load Successfully Sent...")


    def send_conc_request(self, username_idx):
        TotalReq.increment()
        username = user_list[username_idx % LIST_SIZE]

        srv = Client(server=self.server_ip, secret=bytes(self.secret_pwd, encoding="ascii"),
                     dict=dict_obj)

        # create request
        req = srv.CreateAuthPacket(code=pyrad.packet.AccessRequest, User_Name=username,
                                   NAS_Identifier="Creator-VM", NAS_IP_Address="10.212.10.211")

        req["User-Password"] = req.PwCrypt("test")

        reply = srv.SendPacket(req)

        AuthReqSent.increment()

        if reply is None:
            AuthReqError.increment()
            return

        if reply.code == pyrad.packet.AccessAccept:
            AuthReqAccept.increment()
            return

        if reply.code == pyrad.packet.AccessTimeout:
            AuthReqTimeout.increment()
            return

        AuthReqReject.increment()
        return

    def create_threads(self):
        try:
            with ThreadPoolExecutor(max_workers=self.concurrency) as executor:
                executor.map(self.send_conc_request, self.user_idx_list, timeout=12)

        except Exception as exc:
            print("Unable to Create Threads Broken ThreadPool", exc)

Answer 1

first of all, threads do not help too much to cpu intensive processes. I think that multiprocessing can be more effective. To solve your problem I suggest the "divide and conquer" approach, if you split your code into smaller pieces, you can measure the performance on each and see the effectiveness of any piece of code. There any many tools that can help you, like profiler, line_profiler or perf_tool (I'm the author) that is born to solve these kind of problems