reqExecutions IBrokers package

Question

reqExecutions IBrokers package

2.4k Views Asked by aajkaltak At 22 February 2016 at 17:14

Will it be possible for someone to provide me a working example of reqExecutions? I am having a hard time with the ewrapper and callback mechanism. After searching google for working examples, I could not get my hands on anything that would simply work. Please note that I am not a programmer and that is why having a hard time getting my head wrapped around ewrapper and callback.

Original Q&A

There are 1 best solutions below

**bgoldst** · Accepted Answer · 2016-02-29T13:19:38.520000

Disclaimer

Before answering this question, I feel I should emphasize the disclaimer given at the very start of the IBrokers documentation:

This software is in no way affiliated, endorsed, or approved by Interactive Brokers or any of its affiliates. It comes with absolutely no warranty and should not be used in actual trading unless the user can read and understand the source.

So it looks like this package is designed and maintained by independent programmers who may or may not have a good tie-in with official IB API development now or in the future.

Further to the above, I looked at a lot of the package source, and it's fairly incomplete, vis-à-vis the actual IB API source. In fact, you've stumbled upon one of the strands of incompleteness; in the IBrokers Reference Card it says:

Executions

Returns execution details in a twsExecution object. This method is currently only implemented as a request, with no built-in mechanism to manage response data apart from it being discarded.

(Note: You can access the reference card with IBrokersRef(), if you've configured your options()$pdfviewer. If not, you can just open the PDF manually; run system.file('doc/IBrokersREFCARD.pdf',package='IBrokers') to get its location.)

So, the bottom line is, be careful with this package; you shouldn't rely on it for real trading unless you really know what you're doing.

Functionality

Taking a look at the actual reqExecutions() package function, we have:

function (twsconn, reqId = "0", ExecutionFilter)
{
    if (!is.twsConnection(twsconn))
        stop("invalid 'twsConnection' object")
    con <- twsconn[[1]]
    VERSION <- "3"
    outgoing <- c(.twsOutgoingMSG$REQ_EXECUTIONS, VERSION, as.character(reqId),
        ExecutionFilter$clientId, ExecutionFilter$acctCode, ExecutionFilter$time,
        ExecutionFilter$symbol, ExecutionFilter$secType, ExecutionFilter$exchange,
        ExecutionFilter$side)
    writeBin(outgoing, con)
}

To summarize the above, it:

Validates the given TWS connection object is of the proper S3 type. If you look into is.twsConnection(), it just checks that it inherits from twsConnection or twsconn. You can create such a connection with twsConnect(). It's a plain R environment internally, classed as twsconn.
Extracts the socket connection which is wrapped by the TWS connection object. They use an unusual design here; they've overloaded the `[[`() function for the twsconn class. If you look into IBrokers:::`[[.twsconn`, you see it just returns the twsconn$conn environment entry.
Packs the request data (properly encoded) onto the socket. The encoded data consists of 10 NUL-separated fields (the NULs are added by writeBin()): a request type enumeration, a request version, a request identifier (which could be used to differentiate multiple simultaneous requests of the same type), and then 7 filter fields. (Unfortunately, it requires the caller to fully specify all filter fields.) It then returns immediately without waiting for a response.

Contrast the above with a fully-implemented request function, reqCurrentTime():

function (twsconn)
{
    .reqCurrentTime(twsconn)
    con <- twsconn[[1]]
    e_current_time <- eWrapper()
    e_current_time$currentTime <- function(curMsg, msg, timestamp,
        file, ...) {
        msg[2]
    }
    while (isConnected(twsconn)) {
        socketSelect(list(con), FALSE, NULL)
        curMsg <- readBin(con, character(), 1)
        currentTime <- processMsg(curMsg, con, eWrapper = e_current_time,
            twsconn = twsconn, timestamp = NULL, file = "")
        if (curMsg == .twsIncomingMSG$CURRENT_TIME)
            break
    }
    structure(as.numeric(currentTime), class = c("POSIXt", "POSIXct"))
}

This:

Delegates to a package-private function .reqCurrentTime() to send the actual request, similar to what reqExecutions() does. I won't include it here, but you can view its body with IBrokers:::.reqCurrentTime.
Generates a list of callback functions using eWrapper() which it strangely names e_current_time.
Overrides the default handler for the response, which will arrive via a call to currentTime() on the callback list object. The handler simply returns the second field, msg[2], which represents the server's idea of the current time, encoded as a Unix epoch value (seconds since 1970-01-01).
Iterates on the socket, waiting for incoming socket data.
1. When a message arrives, it grabs the first byte into curMsg, which is a code representing the message type, and calls processMsg() on it. This is another function provided by the IBrokers package. This is the function that actually switches on the message code and calls the appropriate callback function on e_current_time, passing it curMsg as well as the code-specific trailing fields, decoded via a call to readBin() (not shown above; see processMsg for the code).
2. Grabs the return value of the callback function (recall this is msg[2], the second field after the message code) into currentTime.
3. If the message code was indeed for the current time request, it breaks the while loop. (If it wasn't, then processMsg() actually triggered a default handler which doesn't do anything useful, and the currentTime value would be ignored.)
Builds a POSIXct object around the currentTime value. All that's really required here is classing it as POSIXt and POSIXct, since the POSIXct type is conveniently implemented by storing the Unix epoch time to represent a date/time point.

So, as you can see, reqCurrentTime() is doing a lot more than reqExecutions(). In its current form, reqExecutions() doesn't do anything to process the response, so it's not really useful at all.

Solution

Since I'm familiar with the IB API, I was able to fill in the missing functionality, which I present below.

As a side note, I referenced the C++ API source code which is available from https://www.interactivebrokers.com/en/index.php?f=5041; the official API source can be taken as authoritative with respect to how the client needs to interact with the socket. For example, you can view the EClient::reqExecutions() function in /TWS API/source/CppClient/client/EClient.cpp to see how it encodes request fields onto the socket, and similarly you can view the EDecoder::processExecutionDataMsg() function in the /TWS API/source/CppClient/client/EDecoder.cpp file to see how it decodes the response from the server. Basically, it uses some C preprocessor macros (ENCODE_FIELD() and DECODE_FIELD()) which expand to a call to some C++ template functions for encoding (template<class T> void EClient::EncodeField(std::ostream& os, T value)) and C++ overloaded functions for decoding (bool EDecoder::DecodeField(bool/int/long/double/std::string& doubleValue, const char*& ptr, const char* endPtr)) which ultimately use the C++ streaming operators to stream primitive fields to the socket std::ostream followed by a NUL for encoding, and call atoi(), atof(), or std::string::operator=() for decoding right from the socket buffer.

I also recommend looking into the official API documentation at https://www.interactivebrokers.com/en/software/api/api.htm.

Firstly, notice that IBrokers provides functions like twsContract() and twsOrder() to allow constructing TWS-specific data objects. There's no twsExecution() function, so I wrote my own, following the same style of object construction, including attaching a twsExecution S3 class. I also wrote a print.twsExecution() function so twsExecution objects would print the same way as the others, which basically means str(unclass(x)).

Secondly, I wrote my own replacement for reqExecutions() called reqExecutions2() which encodes the request data onto the socket as per reqExecutions(), and then overrides the response handler and iterates on the socket waiting for response messages, similar to reqCurrentTime(). I was a little bit more verbose with the code, as I tried to follow the C++ algorithm as closely as possible, including its request version checks on response handling to conditionally take certain fields off the socket. I also included monitoring for error messages from the server, so that the while loop automatically breaks and the function returns on errors. reqExecutions2() returns all response records as a list, where each component is a nested list of reqId, contract, and execution components. This follows the execDetails() callback design in the official API; see https://www.interactivebrokers.com/en/software/api/api.htm (C++ -> Class EWrapper Functions -> Executions -> execDetails()).

Lastly, for convenience I wrote a wrapper around reqExecutions2() called reqExecutionsFrame(), which converts the list of records into a single data.frame.

So, without further ado, here's the code:

library(IBrokers);

## constructor for an execution object
twsExecution <- function(
    execId=NA_character_,
    time=NA_character_,
    acctNumber=NA_character_,
    exchange=NA_character_,
    side=NA_character_,
    shares=NA_integer_,
    price=NA_real_,
    permId=NA_integer_,
    clientId=NA_integer_, ## no long type in R, but decoding process squeezes longs through ints, so may as well use int
    orderId=NA_integer_, ## no long type in R, but decoding process squeezes longs through ints, so may as well use int
    liquidation=NA_integer_,
    cumQty=NA_integer_,
    avgPrice=NA_real_,
    orderRef=NA_character_,
    evRule=NA_character_,
    evMultiplier=NA_real_
) {
    structure(
        list(
            execId=execId,
            time=time,
            acctNumber=acctNumber,
            exchange=exchange,
            side=side,
            shares=shares,
            price=price,
            permId=permId,
            clientId=clientId,
            orderId=orderId,
            liquidation=liquidation,
            cumQty=cumQty,
            avgPrice=avgPrice,
            orderRef=orderRef,
            evRule=evRule,
            evMultiplier=evMultiplier
        ),
        class='twsExecution'
    );
}; ## end twsExecution()
print.twsExecution <- function(x,...) str(unclass(x));

## replacement for reqExecutions()
reqExecutions2 <- function(twscon,reqId=0L,filter=list()) {

    ## validate the connection object
    if (!is.twsConnection(twscon)) stop('invalid twsConnection object.');
    if (!isConnected(twscon)) stop('peer has gone away. check your IB connection',call.=F);

    ## shallow validation of args
    if (!is.integer(reqId) || length(reqId) != 1L) stop('reqId must be a scalar integer.');
    if (!is.list(filter) || (is.null(names(filter)) && length(filter) > 0L)) stop('filter must be a named list.');

    ## send encoded request
    socketcon <- twscon[[1]]; ## extract socket connection from TWS connection object
    VERSION <- '3';
    prepareField <- function(x) if (is.null(x) || length(x) != 1L || is.na(x)) '' else as.character(x); ## empty string is accepted as unspecified
    outgoing <- c(
        .twsOutgoingMSG$REQ_EXECUTIONS,
        VERSION,
        prepareField(reqId), ## will receive this in the response along with data
        prepareField(filter$clientId), ## any client id; if invalid, will get zero results
        prepareField(filter$acctCode), ## must be a valid account code; seems to be ignored completely if invalid
        prepareField(filter$time), ## yyyymmdd HH:MM:SS
        prepareField(filter$symbol), ## must be a valid contract symbol, case-insensitive
        prepareField(filter$secType), ## STK|OPT|FUT|IND|FOP|CASH|BAG|NEWS
        prepareField(filter$exchange), ## must be a valid exchange name, case-insensitive; seems to be ignored completely if invalid
        prepareField(filter$side) ## buy|sell
    );
    writeBin(outgoing,socketcon); ## automatically appends a NUL after each vector element

    ## set handler method
    ## note: don't need to explicitly handle execDetailsEnd(); it provides no new data, and the below while-loop will check for and break on it
    ew <- eWrapper();
    ew$execDetails <- function(curMsg,msg,timestamp,file,...) {

        ## reqId and most contract and execution fields are returned in a character vector in msg
        ## build a return value by mapping the fields to their corresponding parameters of twsContract() and twsExecution()
        n <- (function() { n <- 0L; function() n <<- n+1L; })();
        version <- as.integer(msg[n()]);
        reqId <- if (version >= 7L) as.integer(msg[n()]) else -1L;
        orderId <- as.integer(msg[n()]); ## not sure why this is out-of-order with the remaining execution fields
        ## contract fields
        conId <- as.integer(msg[n()]);
        symbol <- msg[n()];
        secType <- msg[n()];
        lastTradeDateOrContractMonth <- msg[n()];
        strike <- as.double(msg[n()]);
        right <- msg[n()];
        multiplier <- ''; ##multiplier <- if (version >= 9L) msg[n()] else ''; ----- missing?
        exch <- msg[n()];
        primaryExchange <- ''; ## not returned
        currency <- msg[n()];
        localSymbol <- msg[n()];
        tradingClass <- if (version >= 10L) msg[n()] else '';
        includeExpired <- F; ## not returned
        secIdType <- ''; ## not returned
        secId <- ''; ## not returned
        comboLegsDescrip <- ''; ## not returned
        comboLegs <- ''; ## not returned
        underComp <- 0L; ## not returned
        ## execution fields
        execId <- msg[n()];
        time <- msg[n()];
        acctNumber <- msg[n()];
        exchange <- msg[n()];
        side <- msg[n()];
        shares <- as.integer(msg[n()]);
        price <- as.double(msg[n()]);
        permId <- as.integer(msg[n()]);
        clientId <- as.integer(msg[n()]);
        ## (orderId already assigned)
        liquidation <- as.integer(msg[n()]);
        cumQty <- if (version >= 6L) as.integer(msg[n()]) else 0L;
        avgPrice <- if (version >= 6L) as.double(msg[n()]) else 0;
        orderRef <- if (version >= 8L) msg[n()] else '';
        evRule <- if (version >= 9L) msg[n()] else '';
        evMultiplier <- if (version >= 9L) as.double(msg[n()]) else 0;

        ## build the list to return
        ## note: the twsContract() and twsExecution() functions provided with the IBrokers package as of 0.9-12 do not take all of the above fields; we'll pass what they take
        list(
            reqId=reqId,
            contract=twsContract(
                conId=conId,
                symbol=symbol,
                sectype=secType,
                exch=exch,
                primary=primaryExchange,
                expiry=lastTradeDateOrContractMonth,
                strike=strike,
                currency=currency,
                right=right,
                local=localSymbol,
                multiplier=multiplier,
                combo_legs_desc=comboLegsDescrip,
                comboleg=comboLegs,
                include_expired=includeExpired,
                secIdType=secIdType,
                secId=secId
            ),
            execution=twsExecution(
                execId=execId,
                time=time,
                acctNumber=acctNumber,
                exchange=exchange,
                side=side,
                shares=shares,
                price=price,
                permId=permId,
                clientId=clientId,
                orderId=orderId,
                liquidation=liquidation,
                cumQty=cumQty,
                avgPrice=avgPrice,
                orderRef=orderRef,
                evRule=evRule,
                evMultiplier=evMultiplier
            )
        );

    }; ## end execDetails()

    ## hack errorMessage() so we can differentiate between true errors and info messages; not the best design on the part of IB to conflate these
    body(ew$errorMessage)[[length(body(ew$errorMessage))+1L]] <- substitute(msg);

    ## iterate until we get the expected responses off the socket
    execList <- list();
    while (isConnected(twscon)) {
        socketSelect(list(socketcon),F,NULL);
        curMsg <- readBin(socketcon,character(),1L);
        res <- processMsg(curMsg,socketcon,eWrapper=ew,twsconn=twscon,timestamp=NULL,file='');
        ## check for error
        if (curMsg == .twsIncomingMSG$ERR_MSG) {
            ## note: the actual message was already catted inside processMsg() -> ew$errorMessage(); just abort if true error
            code <- as.integer(res[3L]);
            if (!code%in%c( ## blacklist info messages
                0   , ## "Warning: Approaching max rate of 50 messages per second (%d)"
                2103, ## "A market data farm is disconnected."
                2104, ## "A market data farm is connected."
                2105, ## "A historical data farm is disconnected."
                2106, ## "A historical data farm is connected."
                2107, ## "A historical data farm connection has become inactive but should be available upon demand."
                2108, ## "A market data farm connection has become inactive but should be available upon demand."
                2119  ## "Market data farm is connecting:%s" -- undocumented
            )) stop(paste0('request error ',code));
        }; ## end if
        ## check for data
        if (curMsg == .twsIncomingMSG$EXECUTION_DATA)
            execList[[length(execList)+1L]] <- res;
        ## check for completion
        if (curMsg == .twsIncomingMSG$EXECUTION_DATA_END) break;
    }; ## end while

    execList;

}; ## end reqExecutions2()

reqExecutionsFrame <- function(...) {
    res <- reqExecutions2(...);
    do.call(rbind,lapply(res,function(e) do.call(data.frame,c(list(reqId=e$reqId),e$contract,e$execution,stringsAsFactors=F))));
}; ## end reqExecutionsFrame()

Here's a demo on my paper trading account:

## create the TWS connection, selecting an arbitrary client id
twscon <- twsConnect(0L);

twscon; ## this is how it displays by default
## <twsConnection,0 @ 20160229 07:36:33 EST, nextId=4268>
(function(x) c(typeof(x),mode(x),class(x)))(twscon); ## show type info
## [1] "environment" "environment" "twsconn"     "environment"
ls(twscon); ## list the entries in the environment
## [1] "clientId" "conn" "connected" "connected.at" "nextValidId" "port" "server.version"
twscon$conn; ## actual socket connection across which I/O travels between the client and server
##        description              class               mode               text
## "->localhost:7496"         "sockconn"               "ab"           "binary"
##             opened           can read          can write
##           "opened"              "yes"              "yes"

## demo the current time request
## note some info messages are always written onto the socket by the server after we create a connection; the while loop simply passes through them before getting to the current time response
reqCurrentTime(twscon);
## TWS Message: 2 -1 2104 Market data farm connection is OK:cashfarm
## TWS Message: 2 -1 2104 Market data farm connection is OK:usfarm
## TWS Message: 2 -1 2106 HMDS data farm connection is OK:cashhmds
## TWS Message: 2 -1 2106 HMDS data farm connection is OK:ushmds
## [1] "2016-02-29 07:40:10 EST"

## demo the executions request code; shows some random executions I did earlier today (in a paper trading account, of course!)
reqExecutionsFrame(twscon);
##   reqId    conId symbol sectype     exch primary expiry strike currency right   local multiplier combo_legs_desc comboleg include_expired secIdType secId                  execId               time acctNumber exchange side shares   price    permId clientId    orderId liquidation cumQty avgPrice orderRef evRule evMultiplier
## 1     0 15016062    USD    CASH IDEALPRO                     0      CAD       USD.CAD                                               FALSE                 0001f4e8.56d38c6c.01.01 20160229  02:58:06   XXXXXXXX IDEALPRO  SLD 100000 1.35305 195295721        0 2147483647           0 100000  1.35305     <NA>   <NA>           NA
## 2     0 15016062    USD    CASH IDEALPRO                     0      CAD       USD.CAD                                               FALSE                 0001f4e8.56d38c6f.01.01 20160229  02:58:15   XXXXXXXX IDEALPRO  BOT  25000 1.35310 195295723        0 2147483647           0  25000  1.35310     <NA>   <NA>           NA
## 3     0 15016062    USD    CASH IDEALPRO                     0      CAD       USD.CAD                                               FALSE                 0001f4e8.56d38c76.01.01 20160229  02:58:42   XXXXXXXX IDEALPRO  BOT  75000 1.35330 195295723        0 2147483647           0 100000  1.35325     <NA>   <NA>           NA
## 4     0 15016062    USD    CASH IDEALPRO                     0      CAD       USD.CAD                                               FALSE                 0001f4e8.56d39e0b.01.01 20160229  05:48:50   XXXXXXXX IDEALPRO  BOT 100000 1.35710 195295940        0 2147483647           0 100000  1.35710     <NA>   <NA>           NA
## 5     0 15016062    USD    CASH IDEALPRO                     0      CAD       USD.CAD                                               FALSE                 0001f4e8.56d39e16.01.01 20160229  05:49:14   XXXXXXXX IDEALPRO  SLD 100000 1.35720 195295942        0 2147483647           0 100000  1.35720     <NA>   <NA>           NA

## demo some filtering
reqExecutionsFrame(twscon,filter=twsExecutionFilter(side='buy'));
##   reqId    conId symbol sectype     exch primary expiry strike currency right   local multiplier combo_legs_desc comboleg include_expired secIdType secId                  execId               time acctNumber exchange side shares  price    permId clientId    orderId liquidation cumQty avgPrice orderRef evRule evMultiplier
## 1     0 15016062    USD    CASH IDEALPRO                     0      CAD       USD.CAD                                               FALSE                 0001f4e8.56d38c6f.01.01 20160229  02:58:15   XXXXXXXX IDEALPRO  BOT  25000 1.3531 195295723        0 2147483647           0  25000  1.35310     <NA>   <NA>           NA
## 2     0 15016062    USD    CASH IDEALPRO                     0      CAD       USD.CAD                                               FALSE                 0001f4e8.56d38c76.01.01 20160229  02:58:42   XXXXXXXX IDEALPRO  BOT  75000 1.3533 195295723        0 2147483647           0 100000  1.35325     <NA>   <NA>           NA
## 3     0 15016062    USD    CASH IDEALPRO                     0      CAD       USD.CAD                                               FALSE                 0001f4e8.56d39e0b.01.01 20160229  05:48:50   XXXXXXXX IDEALPRO  BOT 100000 1.3571 195295940        0 2147483647           0 100000  1.35710     <NA>   <NA>           NA
reqExecutionsFrame(twscon,filter=twsExecutionFilter(side='buy',time='20160229 04:00:00'));
##   reqId    conId symbol sectype     exch primary expiry strike currency right   local multiplier combo_legs_desc comboleg include_expired secIdType secId                  execId               time acctNumber exchange side shares  price    permId clientId    orderId liquidation cumQty avgPrice orderRef evRule evMultiplier
## 1     0 15016062    USD    CASH IDEALPRO                     0      CAD       USD.CAD                                               FALSE                 0001f4e8.56d39e0b.01.01 20160229  05:48:50   XXXXXXXX IDEALPRO  BOT 100000 1.3571 195295940        0 2147483647           0 100000   1.3571     <NA>   <NA>           NA

## demo error handling
reqExecutionsFrame(twscon,filter=twsExecutionFilter(side='invalid'));
## TWS Message: 2 0 321 Error validating request:-'gf' : cause - Invalid side
## Error in reqExecutions2(...) : request error 321

reqExecutions IBrokers package

There are 1 best solutions below

Disclaimer

Functionality

Solution

Related Questions in R

Related Questions in INTERACTIVE-BROKERS

Related Questions in IBROKERS

Trending Questions

Popular # Hahtags

Popular Questions