Problem
I am ingesting TI to Azure Sentinel using ThreatConnect and every time the existing IOC is updated, it generates an additional record with modified fields and also shows old records when querying. For example, I created an IOC with a confidence score of 90 and ingested it, later I updated the confidence score to 80 and reingested it. Now if I query it in log analytics it shows both IOCs with confidence scores of 90 and 80 rather than updating the old values. Old records show up until the retention period.
I tried all methods for ThreatIntellegence ingestion - 'TI data connector', 'Graph API', and 'TI preview' tab and got duplicates regardless of the source/method.
Steps to reproduce the behavior
- Add IOC in Azure Sentinel via data connector, Graph API, or TI preview tab.
- Update IOC and re-ingest.
- Query the IOC and it will show both old and updated IOC.
Expected behavior After updating the old IOC, there should only be the updated IOC and single entry.
Has anyone faced the same issue or knows how to resolve this problem?
Thanks in advance for your help!
This is by design. Log Analytics tables are immutable and data can only be added. The pattern use here, and in similar situations is to use arg_max:
ThreatIntelligenceIndicator | where ExpirationDateTime >= now() | summarize arg_max(TimeGenerated, *) by IndicatorId