Hi, I am running alloy on a host and have alloy configured to send journald logs to grafana and filtering by the systemd unit. Everything works fine and I can query the logs in grafana cloud.
However, what I find strange is that when I look at Explore > Logs - unless there was a very recent log, the UI just shows me this “no service match” error even if I select a time range (24h) where I know there are logs entries:
But if I query the logs directly, the entries are shown in the result
For the purposes of illustrating the issue, I generated a log entry from my app as I am typing this post and now when I go back to Explore > Log and set the time range to 6h, the latest entry AND the ones from earlier this morning shows up
This behavior seems strange to me. I would expect that when I explore the logs, depending on what time range I select, it should show me all logs for that time range but right now, it only shows me logs only if there is a recent log entry emitted.
My use case here is most likely unique, given that my app does not generate logs as frequently as a typical app would so I’m probably hitting some edge case.
Also just confirmed that this happens if there are no logs emitted for an hour. If the last log emitted was 55 minutes ago and I select the last 15 minutes as the time range, it would just show me “no data” but it still recognizes the service.
Past an hour, I get the “No service matched your search”.
So it seems like to me perhaps log explorer assumes any service that has not had a log entry emitted in the last hour, perhaps it’s considered an inactive service and no longer available from that UI.
Explore Logs is using Loki’s volume endpoint to discover services, so yes, if your service has not had logs during the selected time period, it will not show up. The edge case here is that since the volume endpoint uses metadata from the index, if your service has any chunk within the time range, it will return data. So, in your example, the log from 55 min ago goes in a chunk that covers a 2 hr period ending in 65 min. If you query the last 15m, that chunk is in the range, so it will find the service. However, once you’ve moved past the bounds of that chunk, it won’t.
