How to extract individual and unique live trades from Bitmex Websocket API ws.recent_trades() log

Question

I am trying to stream live trade data from Bitmex to perform some calculations and to automate my trade. I have used the following code obtained from https://github.com/BitMEX/api-connectors/blob/master/official-ws/python/main.py and my code is as follows:

from bitmex_websocket import BitMEXWebsocket
import logging

# Basic use of websocket.
def run():
    logger = setup_logger()

    # Instantiating the WS will make it connect. Be sure to add your api_key/api_secret.
    ws = BitMEXWebsocket(endpoint="https://testnet.bitmex.com/api/v1", symbol="XBTUSD",
                         api_key=api_key, api_secret=api_secret)

    logger.info("Instrument data: %s" % ws.get_instrument())

    # Run forever
    while(ws.ws.sock.connected):
        # CODE TO STREAM THE LIVE TRADE
        logger.info("Recent Trades: %s\n\n" % ws.recent_trades())

def setup_logger():
    # Prints logger info to terminal
    logger = logging.getLogger()
    logger.setLevel(logging.INFO)  # Change this to DEBUG if you want a lot more info
    ch = logging.StreamHandler()
    # create formatter
    formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
    # add formatter to ch
    ch.setFormatter(formatter)
    logger.addHandler(ch)
    return logger

if __name__ == "__main__":
    run()

This is the output I have obtained:

2019-09-08 02:35:35,220 - root - INFO - Recent Trades: [{'timestamp': '2019-09-07T18:35:21.333Z', 'symbol': 'XBTUSD', 'side': 'Sell', 'size': 100, 'price': 10483, 'tickDirection': 'ZeroMinusTick', 'trdMatchID': 'b5f7a502-9d28-5139-19e3-b713f7d86426', 'grossValue': 953900, 'homeNotional': 0.009539, 'foreignNotional': 100}]


2019-09-08 02:35:35,221 - root - INFO - Recent Trades: [{'timestamp': '2019-09-07T18:35:21.333Z', 'symbol': 'XBTUSD', 'side': 'Sell', 'size': 100, 'price': 10483, 'tickDirection': 'ZeroMinusTick', 'trdMatchID': 'b5f7a502-9d28-5139-19e3-b713f7d86426', 'grossValue': 953900, 'homeNotional': 0.009539, 'foreignNotional': 100}]


2019-09-08 02:35:35,222 - root - INFO - Recent Trades: [{'timestamp': '2019-09-07T18:35:21.333Z', 'symbol': 'XBTUSD', 'side': 'Sell', 'size': 100, 'price': 10483, 'tickDirection': 'ZeroMinusTick', 'trdMatchID': 'b5f7a502-9d28-5139-19e3-b713f7d86426', 'grossValue': 953900, 'homeNotional': 0.009539, 'foreignNotional': 100}]

As you can see, all 3 output are for the same trade, how can I get the code to print only live unique trades that happen but not continuously the same trade?

At times, I obtain long chunk of stream which contains a few trades per stream such as:

2019-09-08 02:36:03,539 - root - INFO - Recent Trades: [{'timestamp': '2019-09-07T18:35:21.333Z', 'symbol': 'XBTUSD', 'side': 'Sell', 'size': 100, 'price': 10483, 'tickDirection': 'ZeroMinusTick', 'trdMatchID': 'b5f7a502-9d28-5139-19e3-b713f7d86426', 'grossValue': 953900, 'homeNotional': 0.009539, 'foreignNotional': 100}, {'timestamp': '2019-09-07T18:36:04.168Z', 'symbol': 'XBTUSD', 'side': 'Sell', 'size': 1485, 'price': 10483, 'tickDirection': 'ZeroMinusTick', 'trdMatchID': '8edc4253-85fa-dc93-23e4-3394be2153cc', 'grossValue': 14165415, 'homeNotional': 0.14165415, 'foreignNotional': 1485}, {'timestamp': '2019-09-07T18:36:04.168Z', 'symbol': 'XBTUSD', 'side': 'Sell', 'size': 10, 'price': 10483, 'tickDirection': 'ZeroMinusTick', 'trdMatchID': '09bea4d1-14e2-86af-7152-38d468e7fbed', 'grossValue': 95390, 'homeNotional': 0.0009539, 'foreignNotional': 10},

How can I split them up into individual trades? My desire output will be each individual trade in a list: [timestamp, price, qty, side]

and run a function to perform some calculations on each trade. Thank you!

Answer 1

Good question - it doesn't look like there's an obvious way of getting event-driven websocket messages using that library (even if you try to only subscribe to the trade channel!)

This is the easiest solution given your current code. It keeps note of your last trade, and only logs if there's been a change.

from bitmex_websocket import BitMEXWebsocket
import logging
import time

# Basic use of websocket.
def run():
    logger = setup_logger()

    # Instantiating the WS will make it connect. Be sure to add your api_key/api_secret.
    ws = BitMEXWebsocket(endpoint="https://testnet.bitmex.com/api/v1", symbol="XBTUSD",
                         api_key=api_key, api_secret=api_secret)

    logger.info("Instrument data: %s" % ws.get_instrument())

    # Run forever
    start_time = time.time()
    trades = []

    while(ws.ws.sock.connected):
        recent = ws.recent_trades()[-1] # take only last (most recent) trade

        if not len(trades) or trades[-1] != recent:
            # only store first trade or changed ones
            logger.info("Trade: %s" % recent)
            trades.append(recent)
            # you could call some other useful function here
        else:
            logger.info('Unchanged')

        # below recommended to slow down your poll slightly
        time.sleep(0.05)

        # below is just to exit loop after 5 secs and print the trades
        if time.time() > start_time + 5:
            print(trades)
            break

Output:

2019-09-11 23:57:28,601 - bitmex_websocket - INFO - Connected to WS.
2019-09-11 23:57:30,139 - bitmex_websocket - INFO - Got all market data. Starting.
2019-09-11 23:57:30,140 - root - INFO - Instrument data: {'symbol': 'XBTUSD', ...
2019-09-11 23:57:30,142 - root - INFO - Trade: {'timestamp': '2019-09-11T22:57:29.880Z', 'symbol': 'XBTUSD', 'side': 'Sell', 'size': 159, 'price': 10147, 'tickDirection': 'ZeroMinusTick', 'trdMatchID': 'cc39257d-dc11-5b90-a0cc-ebabe7b6d104', 'grossValue': 1566945, 'homeNotional': 0.01566945, 'foreignNotional': 159}
2019-09-11 23:57:30,196 - root - INFO - Unchanged
2019-09-11 23:57:30,248 - root - INFO - Unchanged
2019-09-11 23:57:30,299 - root - INFO - Unchanged
2019-09-11 23:57:34,614 - root - INFO - Unchanged
2019-09-11 23:57:34,683 - root - INFO - Trade: {'timestamp': '2019-09-11T22:57:33.298Z', 'symbol': 'XBTUSD', 'side': 'Sell', 'size': 145, 'price': 10147, 'tickDirection': 'ZeroMinusTick', 'trdMatchID': '3f12ae9f-371e-6261-3380-456b3c0a3c06', 'grossValue': 1428975, 'homeNotional': 0.01428975, 'foreignNotional': 145}
2019-09-11 23:57:34,752 - root - INFO - Unchanged

[
    {'timestamp': '2019-09-11T22:57:29.880Z', 'symbol': 'XBTUSD', 'side': 'Sell', 'size': 159, 'price': 10147, 'tickDirection': 'ZeroMinusTick', 'trdMatchID': 'cc39257d-dc11-5b90-a0cc-ebabe7b6d104', 'grossValue': 1566945, 'homeNotional': 0.01566945, 'foreignNotional': 159}, 
    {'timestamp': '2019-09-11T22:57:33.298Z', 'symbol': 'XBTUSD', 'side': 'Sell', 'size': 145, 'price': 10147, 'tickDirection': 'ZeroMinusTick', 'trdMatchID': '3f12ae9f-371e-6261-3380-456b3c0a3c06', 'grossValue': 1428975, 'homeNotional': 0.01428975, 'foreignNotional': 145}
]

Hope this helps!

Edit: To extract just the information you want, simply do:

    # Run forever
    start_time = time.time()
    trades = []

    while(ws.ws.sock.connected):
        recent = ws.recent_trades()[-1] # take only last (most recent) trade

        if not len(trades) or trades[-1] != recent:
            # only store first trade or changed ones
            useful = {
                "time": recent["timestamp"],
                "price": recent["price"],
                "size": recent["size"],
            }
            logger.info("Trade: %s" % useful)
            trades.append(useful)

            my_processing_func(trades) 

        # below recommended to slow down your poll slightly
        time.sleep(0.05)

def my_processing_func(trades):
    # This gets called every new trade, and gives you all past trades
    # You may want to write them all to a file or just pass the most recent trade

    just_sizes = [t["size"] for t in trades]

If you want to let it run continuously, remove the block I added:

        # below is just to exit loop after 5 secs and print the trades
        if time.time() > start_time + 5:
            print(trades)
            break