Python serial communication causing high CPU Usage when baudrate is 1000000

Question

I am writing a script using pyserial to communicate with the capture card. The baud rate is 1000000. Now I can receive data normally, but the CPU usage is too high by more than 50%. The following is my main code. Is there any solution?

This is the serial initialization part.

try:
        ser = serial.Serial(
            port = ser_port,
            baudrate=1000000,
            timeout=0.001
        )
        if ser.is_open:
            print("\033[32m [LOG]Serial is open\033[0m")
        else:
            print("\033[31m [ERROR]Serial not Open \033[0m")
    except KeyboardInterrupt as e:
        print("\033[31m [ERROR]Serial not Open | {}\033[0m".format(e))

This is my main loop for my script.

while not rospy.is_shutdown():
        data = ser.read(1)
        if len(data) > 0:
            CmdPacket_Data_Buffer[0:-1] = CmdPacket_Data_Buffer[1:]
            CmdPacket_Data_Buffer[-1:] = data
        
        if CmdPacket_Data_Buffer[-4:] == bytearray([0,0,0x80,0x7F]):
            CmdPacket_Data_Buffer[-1] = 0x77
            f_data = np.frombuffer(CmdPacket_Data_Buffer[:-4],dtype=np.dtype('<f4'))
            counter += 1
            # print(int(round(time.time() * 1000))-t0)# 2ms
            if counter % 5 == 0:
                counter = 0
                f_vec = fp_calib_mat[:,0]+fp_calib_mat[:,1]*f_data**2
                f_vec[f_vec<0] = 0
                fp_buffer[0] = np.sum(f_vec)
                print("\r"+buffer_name+"f_sum={}".format(fp_buffer[0]), end='')
                # print(int(round(time.time() * 1000))-t0)# 10ms