USB between Teensy and Windows PC speed improvements

Question

I am working on a project that involves a PC app (python, QT) is communicating over USB to a Teensy 4.1 and is taking longer than I'd expect to talk.

Right now I estimate the travel time is somewhere > 1ms to transfer 5 bytes, despite using the USB protocol which should be transferring extremely fast.

my connection in the PC app looks like (baud rate should be ignored with USB protocol):

ser_handle = serial.Serial(port, baudrate=9600, writeTimeout=1, timeout=3)

and it writes like:

self.ser_handle.write("1234\n".encode())

On the Teensy side, we read serial like so:

void Serial_Receive() {
  char incoming_byte;  
  while (Serial.available() > 0) {
    incoming_byte = Serial.read();
    rx_buffer.concat(incoming_byte);
    if (incoming_byte == '\n') {
       stringlen = rx_buffer.length();
    }    
  }
}

Once we get a newline char, we will exit this function and process the data. Assuming our loop speed is extremely fast, does this make sense?

Basically looking for any glaring issues that might be the cause of really slow data speeds. Are there any drivers/protocols/setup that needs to be done have USB serial work faster between the PC and teensy?