How to stream data in KDB?

Question

I have access to a realtime KDB server that has tables with new data arriving every millisecond.

Currently, I'm just using a naive method which is basically like:

.z.ts:{
    newData: getNewData[];   / get data arriving in the last second
    data::data uj newData;
};
\t 100;

to ensure that my data (named data) is constantly updated.

However, the uj is very slow (probably due to constant reallocation of memory) and polling is just plain awkward.

I've heard KDB is meant to be good at handling this kind of streaming tick data, so is there a better way? Perhaps some push-based method without need for uj?

Answer 1

Rather than polling. Use kdb+tick the publish subscriber architecture for kdb+.

• Official Manual: https://github.com/KxSystems/kdb/blob/master/d/tick.htm
• kdb+ Tick overview: http://www.timestored.com/kdb-guides/kdb-tick-data-store
• Source code: https://github.com/KxSystems/kdb-tick

Answer 2

If there is a realtime, presumably there's a tickerplant feeding it. You can subscribe to the tickerplant:

.u.sub[`;`];

That means sub all tables, all symbols. The result of the call is an array where 0th element is the table name and 1th element is the current data the tickerplant holds for that table (usually empty or a small number of rows). The tickerplant will then cache the handle to your kdb instance and keep sending it data. BUT it assumes there is a upd function on your kdb instance that can handle the request.

upd:{[t;x] t insert x}

OR

upd:insert

(same thing)

The upd function is called with a table symbol name (t) and the data to insert into it (x).

So a good straightforward implementation overall would be:

upd:insert;
@[`.;:;t:.u.sub[`;`][0];t[1]];  //set result of sub to t, set t to t[1] (initial result)