Is Azure Stream Analytics viable for a use-case of many different sensor types with a single output?

Question

Please me patient with me, I'm a junior developer. I'm working on a project where we're aiming to integrate lots of different LoRaWAN sensors, from multiple vendors. This means we are ingesting many different types of payloads from Azure IoT hub, each with different schemas and measurement types. I want to standardize these different measurements into a single REST API POST request to a cloud service. My project manager wants to use Azure Stream Analytics as a component in this setup, and ingesting messages from IoT Hub and outputting them to the REST API. Incoming sensor messages have different formats and I would need a system to:

1. Read the sensor ID of the message
2. Lookup the sensor ID in an internal sensor database and match it to a known sensor type.
3. Use the matching decoder to pick out the important parts of the received message
4. Post the important bits of information from the message to a POST request

Now to the problem: Is this even doable with the relatively limited query language of Azure Stream Analytics? To me, it sounds like an Azure Function would be a better choice when it comes to these slightly more complex use cases. Or am I misunderstanding the capabilities of the platform completely, and this could be done easily?

I've tried running several SQL commands that could get me closer to a solution, but find myself limited by the "dumbed down" SQL language in ASA.

Answer 1

Azure Stream Analytics is a powerful tool for real-time data stream processing, but it may not be the best fit for all use cases, especially when dealing with complex data transformation.

• I created a Sample Custom logs to register the SensorID, message send, and sensor type. Change the kusto query as per your environment.

• Read the sensor ID of the message:

Azure Stream Analytics can extract fields from incoming messages, so reading the sensor ID is possible.

customEvents
| where name == "SensorDataReceived"
| project customDimensions.SensorID 

• Lookup the sensor ID in an internal sensor database and match it to a known sensor type:

Azure Stream Analytics supports reference data input, but the lookup operation may be limited in complexity. If your sensor database and matching logic are relatively simple, Stream Analytics could handle this, but if it involves complex matching and transformations, an Azure Function or other services might be a better choice.

customEvents
| where name == "SensorDataReceived"
| where customDimensions.SensorType == "TemperatureSensor" and customDimensions.SensorID == "Sensor1"
| project customDimensions.ImportantInfo, customDimensions.SensorType, customDimensions.SensorID

• Use the matching decoder to pick out the important parts of the received message:

This step may require custom code to apply specific decoders based on the sensor type. Azure Stream Analytics allows JavaScript user-defined functions (UDFs), but their capabilities are limited compared to full-fledged Azure Functions.

• Post the important bits of information from the message to a REST API:

Azure Stream Analytics can output data to various destinations, including REST APIs, so it can handle this aspect. The message sent can we register in logs, 'respective properties', the status of get, post, delete, put can we register in logs.