Azure stream analytics how to create a single parquet file

530 Views Asked by At

I have few IoT devices sending telemetry data to Azure Event Hub. I want to write a data to Parquet file in Azure Data Lake so that I can query that data using Azure Synapse.

I have Azure function triggered to Azure event hub, But I did not find a way directly to write a data received from device to Azure data Lake in Parquet format.

So what I am doing, I have Stream Analytics job - which has Input from Event hub and Output to Azure Data lake in Parquet format.

I have configured Stream analytics output path format as different format - but it would create multiple small files within the following folders.

*device-data/{unitNumber}/

device-data/{unitNumber}/{datetime:MM}/{datetime:dd}*

I want to have single parquet file for single device. Can someone help in this?

I have tried to configure Maximum time -> But the data wont get written to parquet file till this time get elapsed. I don't want this as well.

I want simple functionality - as soon as data received from the device to event hub, it should get appended to parquet file in Azure Data lake.

0

There are 0 best solutions below