HW IO and CPU low jitter application

Question

I have a hardware IO task (write/read serial message) that has a strict jitter requirement of less than 200 micro seconds. I need to be able to isolate both a CPU core/s and hardware/interrupt.

I have tried 2 things that have helped but not gotten me all the way there.

1. Using <termios.h> to configure the tty device. Specifically setting VMIN=packet_size and VTIME=0
2. Isolcpus kernel argument in /etc/default/grub and running with taskset

I am still seeing upwards of 5 ms (5000 us) of jitter on my serial reads. I tested this same application with pseudo serial devices (created by socat) to eliminate the HW variable but am still seeing high jitter.

My test application right now just opens a serial connection, configures it, then does a while loop of writes/reads.

Could use advice on how to bring jitter down to 200 us or less. I am considering moving to a dual boot RTOS/Linux with shared memory, but would rather solve on one OS.

Real Application description:

1. Receive message from USB serial
2. Get PTP (precision time protocol) time within 200 us of receiving the first bit
3. Write packet received along with timestamp to shared memory buffer shared with a python application: <timestamp, packet>.
4. Loop.

Also on another isolated HW/core:

1. Read some <timestamp, packet> from a shared memory buffer
2. Poll PTP time until <timestamp>
3. Transmit <packet> at within 200 us of <timestamp> over USB serial
4. Loop

Answer 1

To reduce process latency/jitter:

1. Isolate some cores /etc/default/grub... isolcpus=0

2. Never interrupt RT tasks set /proc/sys/kernel/sched_rt_runtime_us to -1

3. Run high priority on isolated core schedtool -a 0 -F -p 99 -n -20 -e $CMD

To reduce serial latency/jitter

1. File descriptor options O_SYNC
2. Ioctl ASYNC_LOW_LATENCY
3. Termios VMIN = message size and VTIME = 0
4. Use tcdrain after issuing write commands