datacollection program not correctly collecting

Question

I created an data collection app for our company which collect data from our remote devices. The data is collected from a datamailbox which is comparable with an database that works like an 10 day buffer to store the data. this is all correctly working. The data is collected through post api requests. for example :

var url = BuildUrl("syncdata");
                        var response = webClient.CallApi(url, new NameValueCollection() { { "createTransaction","" }, { "lastTransactionId", transactionId } });
                        var data = DynamicJson.Parse(response);
                        transactionId = data.transactionId;

I've been trying to collect multiple devices at one time but the problem is that it starts running and collect the data from the first device which works. Than our second device will start collecting the data but it only starts from where device one ended so i've been losing 12hours of data each run. For performance we use transactionId's.(each set of data has its own Id)

The workflow should be like this :

When the data is retrieved for the first time, the user specifies only the createTransaction filter. The DataMailbox returns all the data of all devices gateways – with historical data – of the account along a transaction ID. For the next calls to the API, the client specifies both createTransaction and lastTransactionId filters. The lastTransactionId is the ID of the transaction that was returned by the latest request. The system returns all the historical data that has been received by the DataMailbox since the last transaction and a new transaction ID. deviceIds is an additional filter on the returned result. You must be cautious when using the combination of lastTransactionId, createTransaction and deviceIds. lastTransactionId is first used to determine what set of data — newer than this transaction ID and from all the Device gateways — must be returned from the DataMailbox, then deviceIds filters this set of data to send data only from the desired device gateways. If a first request is called with lastTransactionId, createTransaction and deviceIds, the following request — implying a new lastTransactionId — does not contain values history from the previous lastTransactionId of the device gateways that were not in the deviceId from previous request.

I'm really struggling with the data collection and have no clue how to use the TransactionId and the LastTransActionId.This is the code for now

try
            {
                CheckLogin();
                using (var webClient = new MyWebClient())
                {
                    bool moreDataAvailable;
                    int samplesCount = 0;
                    string transactionId = Properties.Settings.Default.TransactionId;
                    string lastTransactionId = Properties.Settings.Default.LastTransactionId;
                    do
                    {
                        var url = BuildUrl("syncdata");
                        var response = webClient.CallApi(url, new NameValueCollection() { { "createTransaction","" }, { "lastTransactionId", transactionId } });
                        var data = DynamicJson.Parse(response);
                        transactionId = data.transactionId;
                        var talk2MMessage = getTalk2MMessageHeader(webClient);
                        if (talk2MMessage != null)
                        {
                        }
                        foreach (var ewon in data.ewons)
                        {
                            Directory.CreateDirectory(ewon.name);
                            foreach (var tag in ewon.tags)
                            {
                                try
                                {
                                    Console.WriteLine(Path.Combine(ewon.name, tag.name + ""));
                                    foreach (var sample in tag.history)
                                    {
                                        Console.WriteLine(ewon.name + " " + tag.name + " " + tag.description);
                                        Console.WriteLine(transactionId);
                                        samplesCount++;
                                    }
                                }
                                catch (RuntimeBinderException)
                                {   // Tag has no history. If it's in the transaction, it's most likely because it has alarm history
                                    Console.WriteLine("Tag {0}.{1} has no history.", ewon.name, tag.name);
                                }
                            }
                        }
                        Console.WriteLine("{0} samples written to disk", samplesCount);
                        // Flush data received in this transaction
                        if (Properties.Settings.Default.DeleteData)
                        {
                            //Console.WriteLine("Flushing received data from the DataMailbox...");
                            url = BuildUrl("delete");

                            webClient.CallApi(url, new NameValueCollection() { { "transactionId", transactionId } });
                            Console.WriteLine("DataMailbox flushed.");
                        }
                        //save the transaction id for next run of this program
                        Properties.Settings.Default.LastTransactionId = lastTransactionId;
                        Properties.Settings.Default.Save();
                        // Did we receive all data?
                        try
                        {
                            moreDataAvailable = data.moreDataAvailable;
                        }
                        catch (RuntimeBinderException)
                        {   // The moreDataAvailable flag is not specified in the server response
                            moreDataAvailable = false;
                        }
                        if (moreDataAvailable)
                            Console.WriteLine("There's more data available. Let's get the next part...");
                    }
                    while (moreDataAvailable);

Here are my credentials for starting the collection like all parameters

 static void CheckLogin()
        {
            if (string.IsNullOrEmpty(Properties.Settings.Default.Talk2MDevId))
            {
                Properties.Settings.Default.Talk2MDevId = Prompt("Talk2MDevId");
                Properties.Settings.Default.APIToken = Prompt("API Token");
                string deleteInputString = Prompt("Delete data after synchronization? (yes/no)");
                Properties.Settings.Default.DeleteData = deleteInputString.ToLower().StartsWith("y");
                Properties.Settings.Default.TransactionId = "";
                Properties.Settings.Default.LastTransactionId = "";
                Properties.Settings.Default.Save();
            }

I think it's something with the transactionId and LastTransaction id but i have no clue. More information can be found here: https://developer.ewon.biz/system/files_force/rg-0005-00-en-reference-guide-for-dmweb-api.pdf

Answer 1

As I understand your question, you problem is that for the first few transactionIds, you only get data from device 1 and then only data from device 2.

I'm assuming the following in my answer:

• You didn't specify somewhere else in code the filter on "ewonid"
• When you say you lose 12 hours of data , you are assuming it because "device 2" data are streamed after "device 1" data.
• You did try without the /delete call with no change

/syncdata is an endpoint that returns a block of data for an account since a given transactionId (or oldest block if you didn't provide a transactionID). This data is sorted by storage date by the server, which depends on multiple factors:

• when was the device last "vpn online"
• at which frequency the device is pushing data to datamailbox
• when was that device packet digested by datamailbox service

You could technically have 1 year old data pushed by a device that gets connected back to vpn now, and those data would be registered in the most recent blocks.

For those reasons, the order of data block is not the order of device recording timestamp. You always have to look at the field ewons[].tags[].history[].date to known when that measure was made.

    foreach (var sample in tag.history)
    {
        Console.WriteLine(ewon.name + " " + tag.name + " " + tag.description);
        Console.WriteLine(sample.value + " at " + sample.date);
        Console.WriteLine(transactionId);
        samplesCount++;
    }

In your case, I would assume both devices are configured to push their data once a day, one pushing it's backlog, let's say, at 6AM and the other at 6PM.