handling tcl expect application crash

Question

Is it possible to spawn an application, send commands, expect results, but also 'respawn' the application in case it crashed and continue from the last point? I tried creating procedures for spawning, but I am not able to catch user shell prompt once the application gets closed

Answer 1

So it sounds like you are doing something like telneting or sshing into a shell then running an application. The application dies or hangs so the prompt does not return so expect does not return and so you are stuck. You will need to use timeout to detect a hang and restart your entire process ( including the spawn). Below is the boiler plate I start with when writing expect scripts. The critical thing to help you resolve your problem is to realize that spawn not only sets the spawn_id but also returns the pid of the spawned process. You can use that pid to kill the spawned process if you get an eof and/or a timeout. My boiler plate kills the spawn process on timeout. The timeout does not bail from the expect loop but waits for the eof before exiting. it also accumulates the output of expect command so on timeout you may be able to see where it dies. The boilerplate is in a proc called run. Spawn the process and pass the pid and the spawn id . Reuse the boiler plate to define other procs that will be the steps. put the procs in a script with a counter between them as shown and repeat. The other answerer is correct that the unless the app restarts where you left off you can need to start from scratch. If it does start from where you left off. make the steps granular enough that you know what command to repeat. and step to start from. BoilerPlate

proc run { pid spawn_id buf } {
    upvar $buf buffer;  #buffer to accumulate output of expect
    set bad 0;
    set done 0;
    exp_internal 0; # set to one for extensive debug
    log_user 0; # set to one to watch action
    expect {
        -i $spawn_id
        -re {} {
                append buffer $expect_out(buffer); # accumultate expect output
                exp_continue;
        }
        timeout { 
               send_user "timeout\n"
               append buffer $expect_out(buffer); # accumultate expect output
               exec kill -9 $pid
               set bad 1
               exp_continue;
            }
        fullbuffer {
              send_user " buffer is full\n"
              append buffer $expect_out(buffer); # accumultate expect output
               exp_continue;
            }
        eof { 
                 send_user "Eof detected\n"
                 append buffer $expect_out(buffer); # accumultate expect output
                 set done 1 ;
            }
    }
    set exitstatus [ exp_wait -i $spawn_id ];
    catch { exp_close -i $spawn_id };
    if { $bad } {
        if { $done } {
            throw EXP_TIMEOUT  "Application timeout"
        } 
        throw BAD_ERROR  "unexpected failure "
    }   
    return $exitstatus
}


set count 0
set attempts 0 ; # try 4 times
while { $count == 0  && $attempts < 4 } {
    set buff ""
    set pid [spawn -noecho  ssh user@host ]
    try {
        run $pid $::spawn_id buff
        incr count
        run2 $pid $::spawn_id buff
        incr count
        run3 $pid $::spawn_id buff
        incr count
        run4 $pid $::spawn_id buff
        incr count
    } trap EXP_TIMEOUT { a b } {
        puts "$a $b"
        puts " program failed at step $count"
    } on error { a b } { 
        puts "$a $b"
        puts " program failed at step $count"
    } finally {
        if { $count == 4 } {
            puts "success"
        } else { 
            set count 0
            incr attempts
            puts "$buff" 
            puts "restarting\n"
        } 
    }
}