stream analytics - converting incoming JSON to PowerBI streaming dataset

Question

I have a sensor which is reporting data into the IoTHub in the following format (via a Modbus server):

15/05/2018 14:56:56> Device: [dev], Data:[[{"DisplayName":"Temperature","HwId":"PI-1","Address":"400002","Value":"192","SourceTimestamp":"2018-05-15 13:56:52"},{"DisplayName":"Humidity","HwId":"PI-1","Address":"400001","Value":"397","SourceTimestamp":"2018-05-15 13:56:52"}]]Properties:
'content-type': 'application/edge-modbus-json'

15/05/2018 14:57:00> Device: [dev], Data:[[{"DisplayName":"Temperature","HwId":"PI-1","Address":"400002","Value":"201","SourceTimestamp":"2018-05-15 13:56:57"},{"DisplayName":"Humidity","HwId":"PI-1","Address":"400001","Value":"397","SourceTimestamp":"2018-05-15 13:56:57"}]]Properties:
'content-type': 'application/edge-modbus-json'

15/05/2018 14:57:06> Device: [dev], Data:[[{"DisplayName":"Temperature","HwId":"PI-1","Address":"400002","Value":"201","SourceTimestamp":"2018-05-15 13:57:02"},{"DisplayName":"Humidity","HwId":"PI-1","Address":"400001","Value":"397","SourceTimestamp":"2018-05-15 13:57:02"}]]Properties:
'content-type': 'application/edge-modbus-json'

15/05/2018 14:57:10> Device: [dev], Data:[[{"DisplayName":"Temperature","HwId":"PI-1","Address":"400002","Value":"195","SourceTimestamp":"2018-05-15 13:57:07"},{"DisplayName":"Humidity","HwId":"PI-1","Address":"400001","Value":"397","SourceTimestamp":"2018-05-15 13:57:07"}]]Properties:
'content-type': 'application/edge-modbus-json'

Each sensor is reported within a separate array entry and also split between the Displayname of the sensor and the Value.

What I want is a JSON Payload that I can feed into PowerBI and so this needs to be in the order:

timestamp:time,humidity:humidity_value,temperature:temperature_value

How do I construct a suitable stream analytics query to do this? This input format is typical of Modbus or OPC-UA type devices so will likely come across this a few times.

I tried to use GetArrayElement/(s) but the array has no name in the JSON so there is nothing to reference.

Answer 1

Per my understanding, you could leverage Azure Stream Analytics JavaScript user-defined functions to flat your data.

Assuming that your data looks like this:

{
    Device:"dev01", 
    Data:[
    [
      {"DisplayName":"Temperature","HwId":"PI-1","Address":"400002","Value":"192","SourceTimestamp":"2018-05-15 13:56:52"}
      ,{"DisplayName":"Humidity","HwId":"PI-1","Address":"400001","Value":"397","SourceTimestamp":"2018-05-15 13:56:52"}
     ]
    ]
}   

You could create the following UDFs:

GetValueByPropertyName:

function main(arrs,propertyname) {
    for(var i=0;i<arrs.length;i++){
       var item=arrs[i];
       if(item.hasOwnProperty(propertyname)){
           return item[propertyname];
       }
    }
    return '';
}

GetValueByDisplayName:

function main(dataArr,displayname) {
    if(dataArr){
      for(var j=0;j<dataArr.length;j++)
      {
        var subArr=dataArr[j];
        for(var i=0;i<subArr.length;i++){
          var obj=subArr[i];
          if(obj.DisplayName.toLowerCase()==displayname.toLowerCase())
              return obj.Value;
        }
      }
    }
    return '';
}

Sample query:

select 
input.device as DeviceName,
UDF.GetValueByDisplayName(input.Data,'Temperature') as Temperature,
UDF.GetValueByDisplayName(input.Data,'Humidity') as Humidity,
UDF.GetValueByPropertyName(GetArrayElement(input.Data,0),'SourceTimestamp') as [Timestamp]
from input

TEST:

Answer 2

Actually found a much easier way to handle this directly without functions. Using CASE statement in combination with LAST.

SELECT System.Timestamp as timestamp,

    CASE Address
        WHEN '400001' THEN cast(Value as float)/10 ELSE last(cast(Value as float)/10) over (partition by HwId limit duration(day,1) when Value is not null and Address like '400001')
    END
    AS Humidity,
    CASE Address
        WHEN '400002' THEN cast(Value as float)/10 ELSE last(cast(Value as float)/10) over (partition by HwId limit duration(day,1) when Value is not null and Address like '400002')
    END
    AS Temperature,
    CASE Address
        WHEN '400003' THEN cast(Value as float)/10 ELSE last(cast(Value as float)/10) over (partition by HwId limit duration(day,1) when Value is not null and Address like '400003')
    END
    AS Pressure    
INTO PowerBI
FROM IoTHub as event