High Storage Cost with Azure Time Series Insights Gen2

Question

We are currently confused/concerned by the development of storage cost for the underlying Storage Account of a Time Series Insights Instance.

Time Series Insights Ingress Received Bytes shows an amount of 1.5 GB for the last 30 days:

If I look at the related storage account for the same time span I see an ingress of 7.9 TB:

I know that TSI needs some space for additional indices and so on but this difference is very surprising and will result in a problem when we scale up our solution.

We set the Warm Storage Retention to the maximum of 31 Days.

Is this behavior something to expect when using Time Series Insights or is there anything we can do about this to reduce the usage of the Storage Account? I can not find anything about this behavior in the documentation.

Answer 1

We believe the root case is a mix of the low push rate with the way how TSI appends data to parquet blobs. There is a new algorithm that would fix this issue.

Please, open a support ticket, and then we can fix it.

Thanks.

Answer 2

• With extremely small ingress per partition, TSI makes relatively large writes to Storage by overwriting portions of the files.
• TSI is in the process of deploying an enhancement that will reduce transactions and volume of writes to Storage by many orders of magnitude; especially for small ingress rate.
• As a temporary workaround, for small ingress rate, it is recommended to use a hub with smaller number of partitions, e.g. targeting around 0.2-0.5 MBps per partition.

In future I would prefer raising a support ticket for such investigations