Hello Everyone,
I am using Grafana version 9.3.6. The datasource for Grafana is Elastic Search Logs. Elastic Version is 8.8.1. Both services are running under Red Hat Enterprise Linux edition.
I am trying to create a time series graph using the elastic search logs. The time series graph should display the count of a specific Service Method being called over a period of time.
I have provided 4 Variables in the dashbaord. The variables are query type and each query targets a specific field from the log record.
- Variable : Environment
- Variable: HostName
- Variable: Application Name
- Variable: LogMethod
Each variable runs a query against a specific field in the log record, also each variable value is used in the next variable query for further filtering the data.
EX: When running query for Application Name, it uses the value of Environment variable and Server name variable to display all the available application names in that server and environment.
The issue is mainly on the LogMethod variable, it uses a query and also a Regex filter to display only the available Service Method Names that matches a specific format and display the results as a Drop Down for the user.
Based on the selection of the drop down, the time series graph can plot the number of occurrences for a specific service method over a specific time range. The regex filter it works for few times and displays the method names, sometimes the results are blank. The results are very inconsistent.
Variable: LogMethod
Query: {“find”: “terms”, “field”: “AMSSLog.LogMessage”, “query”: “fields.AMSSLog.Environment: $AmssEnvironment, agent.hostname: $AmssServerList, AMSSLog.ProjectName: $AmssApplication”}
Different Versions of Regex Query :
/.Finished\s([a-zA-Z_]+.[a-zA-Z_]+)./
/.Finished\s(\w+.\w+)./
The regex works well on this online tester https://regex101.com/ for the following data. But it fails in Grafana , sometimes it shows data but mostly None.
Sample Log Data for the field “LogMessage”
Started PG_OEP_OE.pr_bulk_delsert_OE_DETAIL
Finished PG_OEP_OE.pr_bulk_delsert_OE_DETAIL in 43 ms
Finished OracleDBHelper.ExecuteQuery in 47 ms
Building Error list
Errored out recound count is : 0
Finished OutageEventDetailDao.BulkDelsertOutageEventDetails in 49 ms
Outage Event Details Delsert successfully completed in the database
Finished OutageEventProcessingBO.ProcessOutageEvent in 83 ms
Finished OutageEventArrivalMessageHandler.Process in 83 ms
Successfully processed message: CNP.OEP.BusinessEntities.QueueMessageType.OutageEventMessage
Processed message in 106 ms
Finished processing. Sleeping for 00:00:00.0050000 before peeking again.
Listening for messages on queue: .\private$\OEP_EVENT_PUBLISH_Q
A message is available on .\private$\OEP_EVENT_PUBLISH_Q
Processing message on .\private$\OEP_EVENT_PUBLISH_Q
Receving message from .\private$\OEP_EVENT_PUBLISH_Q
Got a timeout while receiving the message. This really shouldn’t happen, but it was most likely already handled by another thread.
Processed message in 1001 ms
Finished processing. Sleeping for 00:00:00.0050000 before peeking again.
Listening for messages on queue: .\private$\OEP_EVENT_PUBLISH_Q
OEPOutageEventArrivalListenerService is running. Processed 2 messages in the last 1 minute. Total messages since start: 2341
OEPOutageEventArrivalListenerService is running. Processed 0 messages in the last 1 minute. Total messages since start: 2341
A message is available on .\private$\OEP_EVENT_PUBLISH_Q
Processing message on .\private$\OEP_EVENT_PUBLISH_Q
Receiving message from .\private$\OEP_EVENT_PUBLISH_Q
Got a timeout while receiving the message. This really shouldn’t happen, but it was most likely already handled by another thread.
Processed message in 1000 ms
Finished processing. Sleeping for 00:00:00.0050000 before peeking again.
Listening for messages on queue: .\private$\OEP_EVENT_PUBLISH_Q
A message is available on .\private$\OEP_EVENT_PUBLISH_Q
Processing message on .\private$\OEP_EVENT_PUBLISH_Q
Receving message from .\private$\OEP_EVENT_PUBLISH_Q
Message received.
Hold Processing is set to False, Processing this message further
Received message: CNP.OEP.BusinessEntities.QueueMessageType.OutageEventMessage, dispatching…
Started OutageEventArrivalMessageHandler.Process
Any suggestion or help is greatly appreciated.
Thanks