A common topic around Log Analytics is ingestion time. How long time does it take before an event is visible in Log Analytics?
The latency depends on three main areas agent time, pipeline time and indexing time. This is all described in this Microsoft Docs article.
In Log Analytics or Kusto, there is a hidden DateTime column in each table called IngestionTime. The time of ingestion is recorded for each record, in that hidden column. The IngestionTime can be used to estimate the end-to-end latency in ingesting data to Log Analytics. TimeGenerated is a timestamp from the source system, for example, a Windows server. By comparing TimeGenerated and IngestionTime we can estimate the latency in getting the data into Log Analytics. More info around IngestionTime policy here.
In the image below a test event is generated on a Windows, note the timestamp (Logged).
When the event is in Log Analytics, we can find it and compare IngestionTime and TimeGenerated. We can see that the difference is around a second. TimeGenerated is the same as “Logged” on the source system. This is just an estimate, as the clocks on the server and in Log Analytics might not be in sync.
If we want to calculate the estimated latency, we can use the following query. It will take all events and estimate the latency in minutes, and order it by latency.
Event | extend LatencyInMinutes = datetime_diff('minute', ingestion_time(), TimeGenerated) | project TimeGenerated, ingestion_time(), LatencyInMinutes | order by LatencyInMinutes
You can also summaries the average latency per hour, and generated a chart, with the following query. This is useful when investigating latency over a longer period of time.
Event | extend LatencyInMinutes = datetime_diff('minute', ingestion_time(), TimeGenerated) | project TimeGenerated, ingestion_time(), LatencyInMinutes | summarize avg(LatencyInMinutes) by bin(TimeGenerated, 1h)
Disclaimer: Cloud is a very fast-moving target. It means that by the time you’re reading this post everything described here could have been changed completely.
Note that this is provided “AS-IS” with no warranties at all. This is not a production-ready solution for your production environment, just an idea, and an example.