VB.Net SerialPort reads are returning fragmented sets of data

Question

The Problem

I have a USB device which creates a Virtual Serial Port on Windows. I am using VB.Net to write and read from the port. My device responds with specific sized set of bytes, but I am finding that SerialPort.Read(byte-array,offset,number-bytes) does not return the full number-bytes but it also does not timeout or generate an exception. Repeated calls return additional fragments (up to 3 calls required). I do not understand why this read method behaves the way it does? Does it think the requested number of bytes is only a suggestion? :-). I would expect that it would wait to fulfill the entire request, unless it timesout first.

Python code using pySerial does not have the same problem.

So, what am I doing wrong here? Am I expecting too much?

Some scenarios are:

1. I write a command to the port and expect to get 4 bytes in response. I get 1 byte first and then 3 bytes on the subsequent call.
2. I write a command and expect 21120 bytes in response. I get 1, 12671 and then 8448 bytes in 3 calls to read from the port.

Here are some excerpts from my code:

Private Sub SetupVirtualSerialPort()
  Dim portName As String = "COM" + (m_DeviceContext * -1).ToString
  Const baud As Int32 = 9600    '7680000
  Const parity As Parity = parity.None
  Const databits As Int32 = 8
  Const stopbits As StopBits = stopbits.One
  m_SerialPort = New SerialPort(portName, baud, parity, databits, stopbits)
  m_SerialPort.WriteTimeout = VSPtimeout
  m_SerialPort.ReadTimeout = VSPtimeout
  m_SerialPort.ReadBufferSize = 2 * RETURN_BUFFER_SIZE
  m_SerialPort.WriteBufferSize = 2 * COMMAND_BUFFER_SIZE
  m_SerialPort.Open()

  ' Register event handlers
  AddHandler m_SerialPort.ErrorReceived, AddressOf m_DriverInterface.Handle_VSP_Error
End Sub


Public Function WriteReadVSPort(ByVal commandLength As Int32, ByVal returnLength As Int32)             As Int32

  Const RetryLimit As Int32 = 5
  Dim NumberRetries As Int32 = 0
  Dim Offset As Int32 = 0
  Dim ExceptionOccurred As Boolean = False
  Dim NumberBytes As Int32 = 0

  Try '  Writing
    m_SerialPort.Write(m_CommandBuffer, 0, commandLength)
  Catch exc As InvalidOperationException
    MessageBox.Show("InvalidOperationException", Application.ProductName)
    ExceptionOccurred = True
  Catch exc As TimeoutException
    MessageBox.Show("TimeoutException", Application.ProductName)
    ExceptionOccurred = True
  End Try

  If Not ExceptionOccurred Then

    Try ' Reading

      ' Working around a problem here: reads are returning fewer 
      ' bytes than requested, though not signalling a timeout exception.
      ' Therefore, we retry if we got fewer bytes than expected, up to five times.
      While NumberRetries < RetryLimit And returnLength > Offset

        NumberBytes = m_SerialPort.Read(m_ReturnBytes, Offset, returnLength - Offset)
        Offset += NumberBytes
        NumberRetries += 1
        If returnLength <> NumberBytes Then
          System.Diagnostics.Debug.Print("Number of bytes read (" & NumberBytes &
            ") not what was requested (" & returnLength & "). Accumulated " & Offset)
        End If

      End While

    Catch exc As InvalidOperationException
      MessageBox.Show("InvalidOperationException", Application.ProductName)
      ExceptionOccurred = True
    Catch exc As TimeoutException
      MessageBox.Show("TimeoutException", Application.ProductName)
      ExceptionOccurred = True
    End Try

  If ExceptionOccurred Then
    Return 1
  Else
    Return 0
  End If

End Function

Thank you.

Answer 1

This is entirely normal for dealing with IO, including streams and ports. Basically, you need to check the returned value and loop. For example:

int offset = 0, read, remaining = ...;
while(remaining > 0 &&
    (read = source.Read(buffer, offset, remaining) > 0)
{
    offset += read;
    remaining -= read;
}
if(remaining > 0) throw new EndOfStreamException();

If your messages are not fixed-length you may need to add a length prefix (before each), or a message delimiter (after each).

Answer 2

I incorporated the suggestion that I use the DataReceived event and make the code event driven rather than looping. I found out the Virtual Serial Port still does not work for my 21120 byte messages in one read operation. Shorter message lengths are done properly. However, when I set the serial port data received threshold to 21119 bytes and set the serial port read buffer to be 10 times as long as my message size, I found that 1. A DataReceived event will be triggered with only 12672 bytes available (not 21119) and the same number returned when a Read() is executed for the full size. 2. With the number of bytes not equaling my threshold, if I do not do a read at that time, no further DataReceived event is triggered 3. But, if (and only if) I read the 12672 bytes, another DataReceived event comes along with the remaining 8448 bytes.

I am clueless as to why this behaves as such. Further comments are welcome.

However, I thought I would share my current code for the benefit of others.

Some class variables are:

Private m_SerialPort As SerialPort = Nothing
Private Debug As Int16
Private m_CommandBuffer(COMMAND_BUFFER_SIZE) As Byte
Private m_ReturnBytes(RETURN_BUFFER_SIZE) As Byte
Private m_WaitingOnBytes As Int32
Private m_VSP_Offset As Int32 = 0
Private m_waitHandle As New System.Threading.ManualResetEvent(True) ' Initialize to signaled
Private m_waitHandle2 As New System.Threading.ManualResetEvent(False) ' Initialize to UN-signaled

Event-handler subroutine

Public Sub Handle_VSP_DataReceived_Dev(ByVal sender As System.Object, ByVal e As System.EventArgs)
  Dim NumberBytes As Int32

  If m_SerialPort.BytesToRead > 0 And m_SerialPort.BytesToRead >= m_WaitingOnBytes Then
    ' This handles the case where the event was triggered, there was data and its length matched
    ' or exceeded the requested amount.
    System.Diagnostics.Debug.Print("DR-Dev: Bytes to read: " & m_SerialPort.BytesToRead & ", waiting for: " & m_WaitingOnBytes)
    NumberBytes = m_SerialPort.Read(m_ReturnBytes, m_VSP_Offset, m_WaitingOnBytes)
    System.Diagnostics.Debug.Print("DR-Dev: got " & NumberBytes & " bytes, released wait handle")
    m_WaitingOnBytes = 0
    m_waitHandle.Set() ' Release the wait handle so the thread running WriteReadVSPort can proceed
  ElseIf m_SerialPort.BytesToRead > 0 And m_WaitingOnBytes > 0 Then
    ' Handle the case where the full request is not delivered. Note: 
    ' This should not need to be done, but it seems that the 
    ' Serial Port is sending the event before all the data is 
    ' received and the threshold is crossed and then not 
    ' sending another event until the buffer is read.
    ' So, here we do a partial read, if we are waiting on a
    ' read operation and adjust the offset and remaining bytes
    ' we are waiting for.
    System.Diagnostics.Debug.Print("DR-Dev: Bytes to read: " & m_SerialPort.BytesToRead & ", waiting for: " & m_WaitingOnBytes)
    NumberBytes = m_SerialPort.Read(m_ReturnBytes, m_VSP_Offset, m_WaitingOnBytes)
    If NumberBytes = m_WaitingOnBytes Then
      ' We actually got all the data, though the serial port did not report it had it ready. Fine, 
      ' proceed as above
      System.Diagnostics.Debug.Print("DR-Dev: got " & m_WaitingOnBytes & " bytes, released wait handle")
      m_WaitingOnBytes = 0
      m_waitHandle.Set() ' Release the wait handle so the thread running WriteReadVSPort can proceed
    Else ' Mark this as a partial read
      System.Diagnostics.Debug.Print("DR-Dev: got partial " & NumberBytes & " while waiting for: " &
        m_WaitingOnBytes & " bytes, continue to hold WriteReadVSPort")
      m_WaitingOnBytes -= NumberBytes
      m_VSP_Offset += NumberBytes
    End If
  End If

End Sub

Function executing the write command/read response activity

Public Function WriteReadVSPort(ByVal commandLength As Int32, ByVal returnLength As Int32) As Int32

  Dim ExceptionOccurred As Boolean = False
  Dim NumberBytes As Int32 = 0
  Dim RetriesRemaining As Int32 = 4
  Dim Finished As Boolean = False

  ' Important to set up for reading response before the command is written
  ' because another thread will handle the DataReceived event and process
  ' the received data without intervention from the thread executing
  ' this(subroutine.
  m_VSP_Offset = 0
  m_WaitingOnBytes = returnLength
  ' Set the DataReceived event threshold
  m_SerialPort.ReceivedBytesThreshold = m_WaitingOnBytes - 1
  ' Set waitHandle so it will block the thread executing this routine until the data is received
  m_waitHandle.Reset()

  Try '  Writing
    m_SerialPort.Write(m_CommandBuffer, 0, commandLength)
  Catch exc As InvalidOperationException
    MessageBox.Show("InvalidOperationException when writing to Serial Port COM" & -1 * DeviceContext, Application.ProductName)
    ExceptionOccurred = True
  Catch exc As TimeoutException
    MessageBox.Show("TimeoutException when writing to Serial Port COM" & -1 * DeviceContext, Application.ProductName)
    ExceptionOccurred = True
  End Try

  If Not ExceptionOccurred Then

    Try ' Reading all done by Event Handler

      ' wait for event handler to complete its job, running in another thread
      System.Diagnostics.Debug.Print("WR_VSP: waiting on waitHandle, bytes avail: " &
              m_SerialPort.BytesToRead & ", want bytes: " & m_WaitingOnBytes)

      If m_waitHandle.WaitOne(VSPtimeout) Then
        ' The WaitOne call returned True, meaning that Handle_VSP_DataReceived_Dev was able to receive all the requested data
        System.Diagnostics.Debug.Print("WR_VSP: proceeding")
      Else
        ' The WaitOne call timed out. Give it some retries before throwing an exception
        While Not Finished And RetriesRemaining > 0
          System.Threading.Thread.Sleep(VSPtimeout)
          If m_SerialPort.BytesToRead > 0 And m_SerialPort.BytesToRead >= m_WaitingOnBytes Then
            NumberBytes = m_SerialPort.Read(m_ReturnBytes, m_VSP_Offset, m_WaitingOnBytes)
            System.Diagnostics.Debug.Print("WR_VSP: timeout mode, got " & m_WaitingOnBytes & " bytes")
            Finished = True
          Else
            RetriesRemaining -= 1
          End If
        End While
        If Not Finished Then
          Throw New TimeoutException("Device failed to send the requested number of bytes.")
        End If
      End If

    Catch exc As InvalidOperationException
      MessageBox.Show("InvalidOperationException when reading from Serial Port COM" & -1 * DeviceContext, Application.ProductName)
      ExceptionOccurred = True
    Catch exc As TimeoutException
      MessageBox.Show("TimeoutException when reading from Serial Port COM" & -1 * DeviceContext, Application.ProductName)
      ExceptionOccurred = True
    End Try

  End If

  If ExceptionOccurred Then
    Return 1
  Else
    Return 0
  End If

End Function