Socket read often return -1 while the buffer is not empty

Question

I am trying to test WiFi data transfer between cell phone and Esp32 (Arduino), when ESP32 reads file data via WiFi, even there is still data in, client.read() often return -1, I have to add other conditions to check reading finished or not.

My question is why there are so many failed reads, any ideas are highly appreciated.

void setup()
{
i=0;
Serial.begin(115200);
Serial.println("begin...");

// You can remove the password parameter if you want the AP to be open.
WiFi.softAP(ssid, password);
IPAddress myIP = WiFi.softAPIP();
Serial.print("AP IP address: ");
Serial.println(myIP);
server.begin();

Serial.println("Server started");
}

// the loop function runs over and over again until power down or reset
void loop()
{

WiFiClient client = server.available();   // listen for incoming clients

if(client)                                // if you get a client,
{
    
    Serial.println("New Client.");           // print a message out the serial port
    Serial.println(client.remoteIP().toString());


    while(client.connected())               // loop while the client's connected
    {
        while(client.available()>0)                // if there's bytes to read from the client,
        {
            char c = client.read();             // read a byte, then
            if(DOWNLOADFILE ==c){

              pretime=millis();
              uint8_t filename[32]={0};
              uint8_t bFilesize[8];
              long filesize;
              int segment=0;
              int remainder=0;
              uint8_t data[512];
              int len=0;
              int totallen=0;
              
              delay(50);
              len=client.read(filename,32);                  
              delay(50);
              len=client.read(bFilesize,8);
              filesize=BytesToLong(bFilesize);
              segment=(int)filesize/512;
              delay(50);

              i=0;  //succeed times
              j=0; //fail times
               ////////////////////////////////////////////////////////////////////
               //problem occures here, to many "-1" return value
               // total read 24941639 bytes, succeed 49725 times, failed 278348 times
               // if there were no read problems, it should only read 48,715 times and finish.
               //But it total read 328,073 times, including 278,348 falied times, wasted too much time
              while(((len=client.read(data,512))!=-1) || (totallen<filesize))
              {
                if(len>-1) {
                  totallen+=len; 
                  i++;
                }
                else{
                  j++;
                }
                               
              }
             ///loop read end , too many times read fail//////////////////////////////////////////////////////////////////
              
              sprintf(toClient, "\nfile name %s,size %d, total read %d, segment %d, succeed %d times, failed %d times\n",filename,filesize,totallen,segment,i,j);
              Serial.write(toClient);
                
              curtime=millis();

              sprintf(toClient, "time splashed %d ms, speed %d Bps\n", curtime-pretime, filesize*1000/(curtime-pretime));
              Serial.write(toClient);
              
              client.write(RETSUCCESS);                  
            }
            else
            {
              Serial.write("Unknow command\n");
            }
        }
    }
           
            
    // close the connection:
    client.stop();
    Serial.println("Client Disconnected.");
}

Answer 1

When you call available() and check for > 0, you are checking to see if there is one or more characters available to read. It will be true if just one character has arrived. You read one character, which is fine, but then you start reading more without stopping to see if there are more available.

TCP doesn't guarantee that if you write 100 characters to a socket that they all arrive at once. They can arrive in arbitrary "chunks" with arbitrary delays. All that's guaranteed is that they will eventually arrive in order (or if that's not possible because of networking issues, the connection will fail.)

In the absence of a blocking read function (I don't know if those exist) you have to do something like what you are doing. You have to read one character at a time and append it to a buffer, gracefully handing the possibility of getting a -1 (the next character isn't here yet, or the connection broke). In general you never want to try to read multiple characters in a single read(buf, len) unless you've just used available() to make sure len characters are actually available. And even that can fail if your buffers are really large. Stick to one-character-at-a-time.

It's a reasonable idea to call delay(1) when available() returns 0. In the places where you try to guess at something like delay(20) before reading a buffer you are rolling the dice - there's no promise that any amount of delay will guarantee bytes get delivered. Example: Maybe a drop of water fell on the chip's antenna and it won't work until the drop evaporates. Data could be delayed for minutes.

I don't know how available() behaves if the connection fails. You might have to do a read() and get back a -1 to diagnose a failed connection. The Arduino documentation is absolutely horrible, so you'll have to experiment.

TCP is much simpler to handle on platforms that have threads, blocking read, select() and other tools to manage data. Having only non-blocking read makes things harder, but there it is.

In some situations UDP is actually a lot simpler - there are more guarantees about getting messages of certain sizes in a single chunk. But of course whole messages can go missing or show up out of order. It's a trade-off.