binance orderbook management

Question

I got orderbook history data of BTCUSDT from the binance api. According to binance api on 'how to manage local orderbook', first get and buffer data using websocket and second get orderbook data using the api and use the lastOrderId to get rid of outdated data that i buffered. however the data i got was 2 csv files. depth_snap and depth_update. so I tried to do what the api told me to do. the first part was already done because it said to get buffer data using websocket which was stored in depth_update. and by using the lastOrderId from the depth_snap I tried to do the second part, remove the outdated data, only to realize that the lastOrderId cannot be used.

I checked the lastOrderId and found that lastOrderId does not overlap in depth_snap and depth_update. so i thought i should use depth_snap data instead of depth_update. however the time gap between data was about 40 minutes which was too long.

I did check timestamp to make sure that the data is in the same date

how can i use this depth_snap and depth_update to create orderbook data? i checked the lastOrderId and pu(lastOrderId of the previous data) between timestamp and csv files(depth_snap.csv on different date) and found that they were in order. will it be okay to just use depth_snap and make orderbook data because the data is continuous?

Answer 1

You should add code to make clear what you have done.

From my own experience, here is how you should use binance websocket:

client = Client('PUBLICKEY')

data = {}

def spread(msg):
    latence = msg['data']['T']-time.time()*1000
    msg['lat'] = latence

    if np.abs(latence) < 500:
        msg['latence'] = False
    else:
        msg['latence'] = True

    with open('data/w'+msg['data']['s']+".dat", 'wb') as out:
        pickle.dump(msg, out)

    os.rename('data/w'+msg['data']['s']+".dat",'data/'+msg['data']['s']+".dat")
    print ("WEBSOCKET %s" % (round(latence)))

bm = BinanceSocketManager(client)
conn_key = bm.start_all_ticker_futures_socket(spread)
bm.start()

In words, you should not be doing any processing in your callback function.

You should write data anywhere you like and process somewhere else from it.

I know those guidelines may seem counter-intuitive, but if not following this I personally get a latency that builds up to dozens of seconds and it stays stuck in the past with no recovery.

You should also compute the latency to flag when to suspend operations.

If you monitor Binance API latency over time, you will observe huge latency peaks some times, especially when you have big moves in the market.