Alerts notifications to Teams are being sent before reaching Repeat interval

Grafana Alerting
,
1

I have alert notifications sent to Teams through a webhook. My understanding is that if the alert status doesn’t change (it remains in it’s alert state), then a notification will not be resent until the “Repeat interval” time. However, what is happening is that the notification is resent when it reaches the “Group interval” time. How do I make Grafana use the “Repeat interval”?
.
The following is my notification policy. I am getting notifications every 2 minutes. I also disabled groups.

image
image716×582 33 KB

Below is a screenshot of the alert’s evaluation time
image

I don’t know if this is relevant but I’m getting the following error in my contact point even though Teams receives notifications.

image
image725×142 9.35 KB

This is in version 10.4.2

2

Show your alert state history and your ms team notifications, Grafana debug logs, pls (formateed pls).

3

Here is a screenshot of the state history and teams. The status hasn’t changed since yesterday at 17:27 which is the last time I triggered the alert.
Please advise how to send the log. I have a file with 10 minutes worth of logs. Also, what is a good way to format? The best I could figure was to replace spaces with newlines. I think it’s more readable but looks odd.

image
image1505×705 108 KB

image
image889×836 43.6 KB

4

Below are the logs which I thought could be relevant. Let me know if there is something missing

logger=tsdb.prometheus
endpoint=queryData
pluginId=prometheus
dsName=prometheus
dsUID=fdtfydy0bhl34f
uname=grafana_scheduler
rule_uid=edyspulkwxmgwc
org_id=1
t=2024-10-11T12:40:02.227357874Z
level=debug
msg="Sending
query"
start=2024-10-11T11:40:00Z
end=2024-10-11T12:40:00Z
step=15s
query="nifi_amount_items_queued
	{component_type=\"Connection\",
	destination_name=\"TFMS_UpdateCounter_acySolaceCounter_T1_P\",
	job=\"nifi\",
	source_name=\"TFMS_ConsumeJMS_NEMS-ACY-Solace_T1_P\"}"
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=A
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:02.229094665Z
	level=debug
	msg="Handling SSE data source query through dataplane"
	datatype=timeseries-multi
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=A
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:02.229194654Z
	level=debug
	msg="Data source queried"
	responseType=dataplane-timeseries-multi
logger=secrets.kvstore
	t=2024-10-11T12:40:02.229257996Z
	level=debug
	msg="got secret value from cache"
	orgId=1
	type=datasource
	namespace=prometheus
logger=tsdb.prometheus
endpoint=queryData
pluginId=prometheus
dsName=prometheus
dsUID=fdtfydy0bhl34f
uname=grafana_scheduler
rule_uid=edyspulkwxmgwc
org_id=1
t=2024-10-11T12:40:02.229380319Z
level=debug
msg="Sending
query"
start=2024-10-11T11:40:00Z
end=2024-10-11T12:40:00Z
step=15s
query="nifi_backpressure_object_threshold
	{component_type=\"Connection\",
	destination_name=\"TFMS_UpdateCounter_acySolaceCounter_T1_P\",
	job=\"nifi\",
	source_name=\"TFMS_ConsumeJMS_NEMS-ACY-Solace_T1_P\"}"
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=B
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:02.230751544Z
	level=debug
	msg="Handling SSE data source query through dataplane"
	datatype=timeseries-multi
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=B
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:02.230834542Z
	level=debug
	msg="Data source queried"
	responseType=dataplane-timeseries-multi
logger=secrets.kvstore
	t=2024-10-11T12:40:02.230879805Z
	level=debug
	msg="got secret value from cache"
	orgId=1
	type=datasource
	namespace=prometheus
logger=tsdb.prometheus
	endpoint=queryData
	pluginId=prometheus
	dsName=prometheus
	dsUID=fdtfydy0bhl34f
	uname=grafana_scheduler
	rule_uid=edyspulkwxmgwc
	org_id=1
	t=2024-10-11T12:40:02.2309948Z
	level=debug
	msg="Sending query"
	start=2024-10-11T11:40:00Z
	end=2024-10-11T12:40:00Z
	step=15s
	query="nifi_amount_items_queued{component_type=\"Connection\",destination_name=\"TFMS_UpdateCounter_oexSolaceCounter_T1_P\",job=\"nifi\",source_name=\"TFMS_
	ConsumeJMS_
	NEMS-OEX-Solace_T1_P\"}"
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=C
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:02.232176817Z
	level=debug
	msg="Handling SSE data source query through dataplane"
	datatype=timeseries-multi
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=C
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:02.232263457Z
	level=debug
	msg="Data source queried"
	responseType=dataplane-timeseries-multi
logger=secrets.kvstore
	t=2024-10-11T12:40:02.232314218Z
	level=debug
	msg="got secret value from cache"
	orgId=1
	type=datasource
	namespace=prometheus
logger=tsdb.prometheus
	endpoint=queryData
	pluginId=prometheus
	dsName=prometheus
	dsUID=fdtfydy0bhl34f
	uname=grafana_scheduler
	rule_uid=edyspulkwxmgwc
	org_id=1
	t=2024-10-11T12:40:02.232457131Z
	level=debug
	msg="Sending query"
	start=2024-10-11T11:40:00Z
	end=2024-10-11T12:40:00Z
	step=15s
	query="nifi_backpressure_object_threshold{component_type=\"Connection\",destination_name=\"TFMS_UpdateCounter_oexSolaceCounter_T1_P\",job=\"nifi\",source_name=\"TFMS_
	ConsumeJMS_
	NEMS-OEX-Solace_T1_P\"}"
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=D
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:02.233902006Z
	level=debug
	msg="Handling SSE data source query through dataplane"
	datatype=timeseries-multi
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=D
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:02.233994046Z
	level=debug
	msg="Data source queried"
	responseType=dataplane-timeseries-multi
logger=ngalert.scheduler
rule_uid=edyspulkwxmgwc
org_id=1
version=149
fingerprint=59c2d878c1a155fe
attempt=1
now=2024-10-11T12:40:00Z
rule_uid=edyspulkwxmgwc
org_id=1
t=2024-10-11T12:40:02.234230187Z
level=debug
msg="Alert rule evaluated"
results="[{Instance:
	State:Alerting
	Error:<nil>
	Results:map[]
	Values:map[ACY Solace queued flowfiles:{Var:ACY Solace queued flowfiles
	Labels:__name__=nifi_amount_items_queued,
	component_id=bdf7406a-c3cd-1910-b74a-903fd1802140,
	component_name=success,
	component_type=Connection,
	destination_id=bdf7406e-c3cd-1910-b3f5-b5c85c0fbc89,
	destination_name=TFMS_UpdateCounter_acySolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74067-c3cd-1910-e4d0-0cff4f72dbc6,
	source_name=TFMS_
	ConsumeJMS_
	NEMS-ACY-Solace_T1_P
	Value:0xc000014ce0}
	ACY backpressure threshold:{Var:ACY backpressure threshold
	Labels:__name__=nifi_backpressure_object_threshold,
	component_id=bdf7406a-c3cd-1910-b74a-903fd1802140,
	component_name=success,
	component_type=Connection,
	destination_id=bdf7406e-c3cd-1910-b3f5-b5c85c0fbc89,
	destination_name=TFMS_UpdateCounter_acySolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74067-c3cd-1910-e4d0-0cff4f72dbc6,
	source_name=TFMS_
	ConsumeJMS_
	NEMS-ACY-Solace_T1_P
	Value:0xc00437ff80}
	OEX Solace queued flowfiles:{Var:OEX Solace queued flowfiles
	Labels:__name__=nifi_amount_items_queued,
	component_id=bdf7406c-c3cd-1910-1552-5be854b97393,
	component_name=success,
	component_type=Connection,
	destination_id=bdf74070-c3cd-1910-cea9-4548fb3d4b2a,
	destination_name=TFMS_UpdateCounter_oexSolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74069-c3cd-1910-d849-580afdf101e9,
	source_name=TFMS_
	ConsumeJMS_
	NEMS-OEX-Solace_T1_P
	Value:0xc00296b260}
	OEX backpressure threshold:{Var:OEX backpressure threshold
	Labels:__name__=nifi_backpressure_object_threshold,
	component_id=bdf7406c-c3cd-1910-1552-5be854b97393,
	component_name=success,
	component_type=Connection,
	destination_id=bdf74070-c3cd-1910-cea9-4548fb3d4b2a,
	destination_name=TFMS_UpdateCounter_oexSolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74069-c3cd-1910-d849-580afdf101e9,
	source_name=TFMS_
	ConsumeJMS_
	NEMS-OEX-Solace_T1_P
	Value:0xc00457e378}
	acyCondition:{Var:acyCondition
	Labels:
	Value:0xc00457e3c8}
	alert:{Var:alert
	Labels:
	Value:0xc00457e458}
	oexCondition:{Var:oexCondition
	Labels:
	Value:0xc00457e418}]
	EvaluatedAt:2024-10-11 12:40:00 +0000 UTC
	EvaluationDuration:2.234186372s
	EvaluationString:[
	var='ACY Solace queued flowfiles'
	labels={__name__=nifi_amount_items_queued,
	component_id=bdf7406a-c3cd-1910-b74a-903fd1802140,
	component_name=success,
	component_type=Connection,
	destination_id=bdf7406e-c3cd-1910-b3f5-b5c85c0fbc89,
	destination_name=TFMS_UpdateCounter_acySolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74067-c3cd-1910-e4d0-0cff4f72dbc6,
	source_name=TFMS_
	ConsumeJMS_
	NEMS-ACY-Solace_T1_P}
	value=100
	],
	[
	var='ACY backpressure threshold'
	labels={__name__=nifi_backpressure_object_threshold,
	component_id=bdf7406a-c3cd-1910-b74a-903fd1802140,
	component_name=success,
	component_type=Connection,
	destination_id=bdf7406e-c3cd-1910-b3f5-b5c85c0fbc89,
	destination_name=TFMS_UpdateCounter_acySolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74067-c3cd-1910-e4d0-0cff4f72dbc6,
	source_name=TFMS_
	ConsumeJMS_
	NEMS-ACY-Solace_T1_P}
	value=100
	],
	[
	var='OEX Solace queued flowfiles'
	labels={__name__=nifi_amount_items_queued,
	component_id=bdf7406c-c3cd-1910-1552-5be854b97393,
	component_name=success,
	component_type=Connection,
	destination_id=bdf74070-c3cd-1910-cea9-4548fb3d4b2a,
	destination_name=TFMS_UpdateCounter_oexSolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74069-c3cd-1910-d849-580afdf101e9,
	source_name=TFMS_
	ConsumeJMS_
	NEMS-OEX-Solace_T1_P}
	value=0
	],
	[
	var='OEX backpressure threshold'
	labels={__name__=nifi_backpressure_object_threshold,
	component_id=bdf7406c-c3cd-1910-1552-5be854b97393,
	component_name=success,
	component_type=Connection,
	destination_id=bdf74070-c3cd-1910-cea9-4548fb3d4b2a,
	destination_name=TFMS_UpdateCounter_oexSolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74069-c3cd-1910-d849-580afdf101e9,
	source_name=TFMS_
	ConsumeJMS_
	NEMS-OEX-Solace_T1_P}
	value=100000
	],
	[
	var='acyCondition'
	labels={}
	value=1
	],
	[
	var='alert'
	labels={}
	value=1
	],
	[
	var='oexCondition'
	labels={}
	value=0
	]}]"
	duration=7.822122ms
logger=ngalert.state.manager
	rule_uid=edyspulkwxmgwc
	org_id=1
	t=2024-10-11T12:40:02.234372921Z
	level=debug
	msg="State manager processing evaluation results"
	resultCount=1
logger=ngalert.state.manager
	rule_uid=edyspulkwxmgwc
	org_id=1
	instance=
	t=2024-10-11T12:40:02.234620326Z
	level=debug
	msg="Setting next state"
	handler=resultAlerting
logger=ngalert.state.manager
	rule_uid=edyspulkwxmgwc
	org_id=1
	instance=
	t=2024-10-11T12:40:02.234665014Z
	level=debug
	msg="Keeping state"
	state=Alerting
	previous_ends_at=2024-10-11T12:41:40Z
	next_ends_at=2024-10-11T12:42:00Z
logger=ngalert.image
	rule_uid=edyspulkwxmgwc
	org_id=1
	dashboard=cdxt4i23o5on4d
	panel=2
	t=2024-10-11T12:40:02.234713444Z
	level=debug
	msg="Requesting
	screenshot"
logger=ngalert.state.manager.persist
	t=2024-10-11T12:40:02.234753733Z
	level=debug
	msg="Saving alert states"
	count=1
	max_state_save_concurrency=1
logger=ngalert.state.manager.persist
	t=2024-10-11T12:40:02.239780301Z
	level=debug
	msg="Saving alert states done"
	count=1
	max_state_save_concurrency=1
	duration=5.023905ms
logger=secrets.kvstore
	t=2024-10-11T12:40:03.338683343Z
	level=debug
	msg="got secret value from cache"
	orgId=1
	type=datasource
	namespace=prometheus

6084
logger=tsdb.prometheus
	endpoint=queryData
	pluginId=prometheus
	dsName=prometheus
	dsUID=fdtfydy0bhl34f
	uname=grafana_scheduler
	rule_uid=edyspulkwxmgwc
	org_id=1
	t=2024-10-11T12:40:23.340817437Z
	level=debug
	msg="Sending query"
	start=2024-10-11T11:40:20Z
	end=2024-10-11T12:40:20Z
	step=15s
	query="nifi_amount_items_queued{component_type=\"Connection\",destination_name=\"TFMS_UpdateCounter_acySolaceCounter_T1_P\",job=\"nifi\",source_name=\"TFMS_ ConsumeJMS_ NEMS-ACY-Solace_T1_P\"}"
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=A
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:23.342332193Z
	level=debug
	msg="Handling SSE data source query through dataplane"
	datatype=timeseries-multi
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=A
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:23.342438058Z
	level=debug
	msg="Data source queried"
	responseType=dataplane-timeseries-multi
logger=secrets.kvstore
	t=2024-10-11T12:40:23.342492059Z
	level=debug
	msg="got secret value from cache"
	orgId=1
	type=datasource
	namespace=prometheus
logger=tsdb.prometheus
	endpoint=queryData
	pluginId=prometheus
	dsName=prometheus
	dsUID=fdtfydy0bhl34f
	uname=grafana_scheduler
	rule_uid=edyspulkwxmgwc
	org_id=1
	t=2024-10-11T12:40:23.342617924Z
	level=debug
	msg="Sending query"
	start=2024-10-11T11:40:20Z
	end=2024-10-11T12:40:20Z
	step=15s
query="nifi_backpressure_object_threshold{component_type=\"Connection\",destination_name=\"TFMS_UpdateCounter_acySolaceCounter_T1_P\",job=\"nifi\",source_name=\"TFMS_ConsumeJMS_NEMS-ACY-Solace_T1_P\"}"
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=B
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:23.34378241Z
	level=debug
	msg="Handling SSE data source query through dataplane"
	datatype=timeseries-multi
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=B
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:23.3438768Z
	level=debug
	msg="Data source queried"
	responseType=dataplane-timeseries-multi
logger=secrets.kvstore
	t=2024-10-11T12:40:23.343923789Z
	level=debug
	msg="got secret value from cache"
	orgId=1
	type=datasource
	namespace=prometheus
logger=tsdb.prometheus
	endpoint=queryData
	pluginId=prometheus
	dsName=prometheus
	dsUID=fdtfydy0bhl34f
	uname=grafana_scheduler
	rule_uid=edyspulkwxmgwc
	org_id=1
	t=2024-10-11T12:40:23.344040967Z
	level=debug
	msg="Sending query"
	start=2024-10-11T11:40:20Z
	end=2024-10-11T12:40:20Z
	step=15s
	query="nifi_amount_items_queued{component_type=\"Connection\",destination_name=\"TFMS_UpdateCounter_oexSolaceCounter_T1_P\",job=\"nifi\",source_name=\"TFMS_ ConsumeJMS_ NEMS-OEX-Solace_T1_P\"}"
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=C
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:23.345281284Z
	level=debug
	msg="Handling SSE data source query through dataplane"
	datatype=timeseries-multi
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=C
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:23.345363099Z
	level=debug
	msg="Data source queried"
	responseType=dataplane-timeseries-multi
logger=secrets.kvstore
	t=2024-10-11T12:40:23.345410784Z
	level=debug
	msg="got secret value from cache"
	orgId=1
	type=datasource
	namespace=prometheus
logger=tsdb.prometheus
	endpoint=queryData
	pluginId=prometheus
	dsName=prometheus
	dsUID=fdtfydy0bhl34f
	uname=grafana_scheduler
	rule_uid=edyspulkwxmgwc
	org_id=1
	t=2024-10-11T12:40:23.345531795Z
	level=debug
	msg="Sending query"
	start=2024-10-11T11:40:20Z
	end=2024-10-11T12:40:20Z
	step=15s
	query="nifi_backpressure_object_threshold{component_type=\"Connection\",destination_name=\"TFMS_UpdateCounter_oexSolaceCounter_T1_P\",job=\"nifi\",source_name=\"TFMS_ ConsumeJMS_ NEMS-OEX-Solace_T1_P\"}"
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=D
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:23.346822699Z
	level=debug
	msg="Handling SSE data source query through dataplane"
	datatype=timeseries-multi
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=D
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:23.346911627Z
	level=debug
	msg="Data source queried"
	responseType=dataplane-timeseries-multi
logger=ngalert.scheduler
	rule_uid=edyspulkwxmgwc
	org_id=1
	version=149
	fingerprint=59c2d878c1a155fe
	attempt=1
	now=2024-10-11T12:40:20Z
	rule_uid=edyspulkwxmgwc
	org_id=1
	t=2024-10-11T12:40:23.347139413Z
	level=debug
	msg="Alert rule evaluated"
	results="[{Instance:
	State:Alerting
	Error:<nil>
	Results:map[]
	Values:map[ACY Solace queued flowfiles:{Var:ACY Solace queued flowfiles
	Labels:__name__=nifi_amount_items_queued,
	component_id=bdf7406a-c3cd-1910-b74a-903fd1802140,
	component_name=success,
	component_type=Connection,
	destination_id=bdf7406e-c3cd-1910-b3f5-b5c85c0fbc89,
	destination_name=TFMS_UpdateCounter_acySolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74067-c3cd-1910-e4d0-0cff4f72dbc6,
	source_name=TFMS_ ConsumeJMS_ NEMS-ACY-Solace_T1_P
	Value:0xc00459b560}
	ACY backpressure threshold:{Var:ACY backpressure threshold
	Labels:__name__=nifi_backpressure_object_threshold,
	component_id=bdf7406a-c3cd-1910-b74a-903fd1802140,
	component_name=success,
	component_type=Connection,
	destination_id=bdf7406e-c3cd-1910-b3f5-b5c85c0fbc89,
	destination_name=TFMS_UpdateCounter_acySolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74067-c3cd-1910-e4d0-0cff4f72dbc6,
	source_name=TFMS_ ConsumeJMS_ NEMS-ACY-Solace_T1_P
	Value:0xc004377240}
	OEX Solace queued flowfiles:{Var:OEX Solace queued flowfiles
	Labels:__name__=nifi_amount_items_queued,
	component_id=bdf7406c-c3cd-1910-1552-5be854b97393,
	component_name=success,
	component_type=Connection,
	destination_id=bdf74070-c3cd-1910-cea9-4548fb3d4b2a,
	destination_name=TFMS_UpdateCounter_oexSolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74069-c3cd-1910-d849-580afdf101e9,
	source_name=TFMS_ ConsumeJMS_ NEMS-OEX-Solace_T1_P
	Value:0xc002db9b80}
	OEX backpressure threshold:{Var:OEX backpressure threshold
	Labels:__name__=nifi_backpressure_object_threshold,
	component_id=bdf7406c-c3cd-1910-1552-5be854b97393,
	component_name=success,
	component_type=Connection,
	destination_id=bdf74070-c3cd-1910-cea9-4548fb3d4b2a,
	destination_name=TFMS_UpdateCounter_oexSolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74069-c3cd-1910-d849-580afdf101e9,
	source_name=TFMS_ ConsumeJMS_ NEMS-OEX-Solace_T1_P
	Value:0xc0044335b8}
	acyCondition:{Var:acyCondition
	Labels:
	Value:0xc004433608}
	alert:{Var:alert
	Labels:
	Value:0xc004433698}
	oexCondition:{Var:oexCondition
	Labels:
	Value:0xc004433658}]
	EvaluatedAt:2024-10-11
	12:40:20 +0000 UTC
	EvaluationDuration:3.347095645s
	EvaluationString:[
	var='ACY Solace queued flowfiles'
	labels={__name__=nifi_amount_items_queued,
	component_id=bdf7406a-c3cd-1910-b74a-903fd1802140,
	component_name=success,
	component_type=Connection,
	destination_id=bdf7406e-c3cd-1910-b3f5-b5c85c0fbc89,
	destination_name=TFMS_UpdateCounter_acySolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74067-c3cd-1910-e4d0-0cff4f72dbc6,
	source_name=TFMS_ ConsumeJMS_ NEMS-ACY-Solace_T1_P}
	value=100
	],
	[
	var='ACY backpressure threshold'
	labels={__name__=nifi_backpressure_object_threshold,
	component_id=bdf7406a-c3cd-1910-b74a-903fd1802140,
	component_name=success,
	component_type=Connection,
	destination_id=bdf7406e-c3cd-1910-b3f5-b5c85c0fbc89,
	destination_name=TFMS_UpdateCounter_acySolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74067-c3cd-1910-e4d0-0cff4f72dbc6,
	source_name=TFMS_ ConsumeJMS_ NEMS-ACY-Solace_T1_P}
	value=100
	],
	[
	var='OEX Solace queued flowfiles'
	labels={__name__=nifi_amount_items_queued,
	component_id=bdf7406c-c3cd-1910-1552-5be854b97393,
	component_name=success,
	component_type=Connection,
	destination_id=bdf74070-c3cd-1910-cea9-4548fb3d4b2a,
	destination_name=TFMS_UpdateCounter_oexSolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74069-c3cd-1910-d849-580afdf101e9,
	source_name=TFMS_ ConsumeJMS_ NEMS-OEX-Solace_T1_P}
	value=0
	],
	[
	var='OEX backpressure threshold'
	labels={__name__=nifi_backpressure_object_threshold,
	component_id=bdf7406c-c3cd-1910-1552-5be854b97393,
	component_name=success,
	component_type=Connection,
	destination_id=bdf74070-c3cd-1910-cea9-4548fb3d4b2a,
	destination_name=TFMS_UpdateCounter_oexSolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74069-c3cd-1910-d849-580afdf101e9,
	source_name=TFMS_ ConsumeJMS_ NEMS-OEX-Solace_T1_P}
	value=100000
	],
	[
	var='acyCondition'
	labels={}
	value=1
	],
	[
	var='alert'
	labels={}
	value=1
	],
	[
	var='oexCondition'
	labels={}
	value=0
	]}]"
	duration=7.33754ms
logger=ngalert.state.manager
	rule_uid=edyspulkwxmgwc
	org_id=1
	t=2024-10-11T12:40:23.347288581Z
	level=debug
	msg="State manager processing evaluation results"
	resultCount=1
logger=ngalert.state.manager
	rule_uid=edyspulkwxmgwc
	org_id=1
	instance=
	t=2024-10-11T12:40:23.347546789Z
	level=debug
	msg="Setting next state"
	handler=resultAlerting
logger=ngalert.state.manager
	rule_uid=edyspulkwxmgwc
	org_id=1
	instance=
	t=2024-10-11T12:40:23.347592119Z
	level=debug
	msg="Keeping state"
	state=Alerting
	previous_ends_at=2024-10-11T12:42:00Z
	next_ends_at=2024-10-11T12:42:20Z
logger=ngalert.image
	rule_uid=edyspulkwxmgwc
	org_id=1
	dashboard=cdxt4i23o5on4d
	panel=2
	t=2024-10-11T12:40:23.347626117Z
	level=debug
	msg="Requesting
	screenshot"
logger=ngalert.state.manager.persist
	t=2024-10-11T12:40:23.347664708Z
	level=debug
	msg="Saving alert states"
	count=1
	max_state_save_concurrency=1
logger=ngalert.state.manager.persist
	t=2024-10-11T12:40:23.366468095Z
	level=debug
	msg="Saving alert states done"
	count=1
	max_state_save_concurrency=1
	duration=18.800583ms
logger=ngalert.sender.router
	rule_uid=edyspulkwxmgwc
	org_id=1
	t=2024-10-11T12:40:23.366558046Z
	level=info
	msg="Sending alerts to local notifier"
	count=1
logger=ngalert.notifier.alertmanager
	org=1
	t=2024-10-11T12:40:23.366597388Z
	level=debug
	component=alertmanager
	orgID=1
	msg="Putting alert"
	alert="Solace reaching or reached backpressure threshold[8f31d10][active]"
	starts_at=2024-10-10T21:27:20Z
	ends_at=2024-10-11T12:42:20Z
logger=ngalert.notifier.alertmanager
	org=1
	t=2024-10-11T12:40:23.366660722Z
	level=debug
	component=alertmanager
	orgID=1
	component=dispatcher
	msg="Received alert"
	alert="Solace reaching or reached backpressure threshold[8f31d10][active]"
logger=secrets.kvstore
	t=2024-10-11T12:40:24.451718451Z
	level=debug
	msg="got secret value from cache"
	orgId=1
	type=datasource
	namespace=prometheus


12864
logger=tsdb.prometheus
	endpoint=queryData
	pluginId=prometheus
	dsName=prometheus
	dsUID=fdtfydy0bhl34f
	uname=grafana_scheduler
	rule_uid=edyspulkwxmgwc
	org_id=1
	t=2024-10-11T12:40:47.780509407Z
	level=debug
	msg="Sending query"
	start=2024-10-11T11:40:40Z
	end=2024-10-11T12:40:40Z
	step=15s
	query="nifi_amount_items_queued{component_type=\"Connection\",destination_name=\"TFMS_UpdateCounter_acySolaceCounter_T1_P\",job=\"nifi\",source_name=\"TFMS_ ConsumeJMS_ NEMS-ACY-Solace_T1_P\"}"
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=A
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:47.78292463Z
	level=debug
	msg="Handling SSE data source query through dataplane"
	datatype=timeseries-multi
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=A
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:47.783022247Z
	level=debug
	msg="Data source queried"
	responseType=dataplane-timeseries-multi
logger=secrets.kvstore
	t=2024-10-11T12:40:47.783071035Z
	level=debug
	msg="got secret value from cache"
	orgId=1
	type=datasource
	namespace=prometheus
logger=tsdb.prometheus
	endpoint=queryData
	pluginId=prometheus
	dsName=prometheus
	dsUID=fdtfydy0bhl34f
	uname=grafana_scheduler
	rule_uid=edyspulkwxmgwc
	org_id=1
	t=2024-10-11T12:40:47.783188426Z
	level=debug
	msg="Sending query"
	start=2024-10-11T11:40:40Z
	end=2024-10-11T12:40:40Z
	step=15s
	query="nifi_backpressure_object_threshold{component_type=\"Connection\",destination_name=\"TFMS_UpdateCounter_acySolaceCounter_T1_P\",job=\"nifi\",source_name=\"TFMS_ ConsumeJMS_ NEMS-ACY-Solace_T1_P\"}"
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=B
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:47.785281938Z
	level=debug
	msg="Handling SSE data source query through dataplane"
	datatype=timeseries-multi
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=B
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:47.78536964Z
	level=debug
	msg="Data source queried"
	responseType=dataplane-timeseries-multi
logger=secrets.kvstore
	t=2024-10-11T12:40:47.785416616Z
	level=debug
	msg="got secret value from cache"
	orgId=1
	type=datasource
	namespace=prometheus
logger=tsdb.prometheus
	endpoint=queryData
	pluginId=prometheus
	dsName=prometheus
	dsUID=fdtfydy0bhl34f
	uname=grafana_scheduler
	rule_uid=edyspulkwxmgwc
	org_id=1
	t=2024-10-11T12:40:47.785536218Z
	level=debug
	msg="Sending query"
	start=2024-10-11T11:40:40Z
	end=2024-10-11T12:40:40Z
	step=15s
	query="nifi_amount_items_queued{component_type=\"Connection\",destination_name=\"TFMS_UpdateCounter_oexSolaceCounter_T1_P\",job=\"nifi\",source_name=\"TFMS_ ConsumeJMS_ NEMS-OEX-Solace_T1_P\"}"
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=C
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:47.787521358Z
	level=debug
	msg="Handling SSE data source query through dataplane"
	datatype=timeseries-multi
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=C
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:47.787688331Z
	level=debug
	msg="Data source queried"
	responseType=dataplane-timeseries-multi
logger=secrets.kvstore
	t=2024-10-11T12:40:47.787797432Z
	level=debug
	msg="got secret value from cache"
	orgId=1
	type=datasource
	namespace=prometheus
logger=tsdb.prometheus
	endpoint=queryData
	pluginId=prometheus
	dsName=prometheus
	dsUID=fdtfydy0bhl34f
	uname=grafana_scheduler
	rule_uid=edyspulkwxmgwc
	org_id=1
	t=2024-10-11T12:40:47.787990934Z
	level=debug
	msg="Sending query"
	start=2024-10-11T11:40:40Z
	end=2024-10-11T12:40:40Z
	step=15s
	query="nifi_backpressure_object_threshold{component_type=\"Connection\",destination_name=\"TFMS_UpdateCounter_oexSolaceCounter_T1_P\",job=\"nifi\",source_name=\"TFMS_ ConsumeJMS_ NEMS-OEX-Solace_T1_P\"}"
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=D
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:47.790177304Z
	level=debug
	msg="Handling SSE data source query through dataplane"
	datatype=timeseries-multi
logger=expr
	rule_uid=edyspulkwxmgwc
	org_id=1
	datasourceType=prometheus
	queryRefId=D
	datasourceUid=fdtfydy0bhl34f
	datasourceVersion=2
	t=2024-10-11T12:40:47.790280073Z
	level=debug
	msg="Data source queried"
	responseType=dataplane-timeseries-multi
logger=ngalert.scheduler
	rule_uid=edyspulkwxmgwc
	org_id=1
	version=149
	fingerprint=59c2d878c1a155fe
	attempt=1
	now=2024-10-11T12:40:40Z
	rule_uid=edyspulkwxmgwc
	org_id=1
	t=2024-10-11T12:40:47.790570492Z
	level=debug
	msg="Alert rule evaluated"
	results="[{Instance:
	State:Alerting
	Error:<nil>
	Results:map[]
	Values:map[ACY Solace queued flowfiles:{Var:ACY Solace queued flowfiles
	Labels:__name__=nifi_amount_items_queued,
	component_id=bdf7406a-c3cd-1910-b74a-903fd1802140,
	component_name=success,
	component_type=Connection,
	destination_id=bdf7406e-c3cd-1910-b3f5-b5c85c0fbc89,
	destination_name=TFMS_UpdateCounter_acySolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74067-c3cd-1910-e4d0-0cff4f72dbc6,
	source_name=TFMS_ ConsumeJMS_ NEMS-ACY-Solace_T1_P
	Value:0xc00458e2b0}
	ACY backpressure threshold:{Var:ACY backpressure threshold
	Labels:__name__=nifi_backpressure_object_threshold,
	component_id=bdf7406a-c3cd-1910-b74a-903fd1802140,
	component_name=success,
	component_type=Connection,
	destination_id=bdf7406e-c3cd-1910-b3f5-b5c85c0fbc89,
	destination_name=TFMS_UpdateCounter_acySolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74067-c3cd-1910-e4d0-0cff4f72dbc6,
	source_name=TFMS_ ConsumeJMS_ NEMS-ACY-Solace_T1_P
	Value:0xc002b6a780}
	OEX Solace queued flowfiles:{Var:OEX Solace queued flowfiles
	Labels:__name__=nifi_amount_items_queued,
	component_id=bdf7406c-c3cd-1910-1552-5be854b97393,
	component_name=success,
	component_type=Connection,
	destination_id=bdf74070-c3cd-1910-cea9-4548fb3d4b2a,
	destination_name=TFMS_UpdateCounter_oexSolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74069-c3cd-1910-d849-580afdf101e9,
	source_name=TFMS_ ConsumeJMS_ NEMS-OEX-Solace_T1_P
	Value:0xc003dbb7d0}
	OEX backpressure threshold:{Var:OEX backpressure threshold
	Labels:__name__=nifi_backpressure_object_threshold,
	component_id=bdf7406c-c3cd-1910-1552-5be854b97393,
	component_name=success,
	component_type=Connection,
	destination_id=bdf74070-c3cd-1910-cea9-4548fb3d4b2a,
	destination_name=TFMS_UpdateCounter_oexSolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74069-c3cd-1910-d849-580afdf101e9,
	source_name=TFMS_ ConsumeJMS_ NEMS-OEX-Solace_T1_P
	Value:0xc0044afa98}
	acyCondition:{Var:acyCondition
	Labels:
	Value:0xc0044afae8}
	alert:{Var:alert
	Labels:
	Value:0xc0044afb78}
	oexCondition:{Var:oexCondition
	Labels:
	Value:0xc0044afb38}]
	EvaluatedAt:2024-10-11 12:40:40	+0000	UTC
	EvaluationDuration:7.790511873s
	EvaluationString:[
	var='ACY Solace queued flowfiles'
	labels={__name__=nifi_amount_items_queued,
	component_id=bdf7406a-c3cd-1910-b74a-903fd1802140,
	component_name=success,
	component_type=Connection,
	destination_id=bdf7406e-c3cd-1910-b3f5-b5c85c0fbc89,
	destination_name=TFMS_UpdateCounter_acySolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74067-c3cd-1910-e4d0-0cff4f72dbc6,
	source_name=TFMS_ ConsumeJMS_ NEMS-ACY-Solace_T1_P}
	value=100
	],
	[
	var='ACY backpressure threshold'
	labels={__name__=nifi_backpressure_object_threshold,
	component_id=bdf7406a-c3cd-1910-b74a-903fd1802140,
	component_name=success,
	component_type=Connection,
	destination_id=bdf7406e-c3cd-1910-b3f5-b5c85c0fbc89,
	destination_name=TFMS_UpdateCounter_acySolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74067-c3cd-1910-e4d0-0cff4f72dbc6,
	source_name=TFMS_ ConsumeJMS_ NEMS-ACY-Solace_T1_P}
	value=100
	],
	[
	var='OEX Solace queued flowfiles'
	labels={__name__=nifi_amount_items_queued,
	component_id=bdf7406c-c3cd-1910-1552-5be854b97393,
	component_name=success,
	component_type=Connection,
	destination_id=bdf74070-c3cd-1910-cea9-4548fb3d4b2a,
	destination_name=TFMS_UpdateCounter_oexSolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74069-c3cd-1910-d849-580afdf101e9,
	source_name=TFMS_ ConsumeJMS_ NEMS-OEX-Solace_T1_P}
	value=0
	],
	[
	var='OEX backpressure threshold'
	labels={__name__=nifi_backpressure_object_threshold,
	component_id=bdf7406c-c3cd-1910-1552-5be854b97393,
	component_name=success,
	component_type=Connection,
	destination_id=bdf74070-c3cd-1910-cea9-4548fb3d4b2a,
	destination_name=TFMS_UpdateCounter_oexSolaceCounter_T1_P,
	exported_instance=https://172.26.21.73:8443/nifi,
	instance=172.26.21.73:28444,
	job=nifi,
	parent_id=a5ee5eaa-305d-3ade-fc86-b7510f49de18,
	source_id=bdf74069-c3cd-1910-d849-580afdf101e9,
	source_name=TFMS_ ConsumeJMS_ NEMS-OEX-Solace_T1_P}
	value=100000
	],
	[
	var='acyCondition'
	labels={}
	value=1
	],
	[
	var='alert'
	labels={}
	value=1
	],
	[
	var='oexCondition'
	labels={}
	value=0
	]}]"
	duration=11.145739ms
logger=ngalert.state.manager
	rule_uid=edyspulkwxmgwc
	org_id=1
	t=2024-10-11T12:40:47.790747624Z
	level=debug
	msg="State manager processing evaluation results"
	resultCount=1
logger=ngalert.state.manager
	rule_uid=edyspulkwxmgwc
	org_id=1
	instance=
	t=2024-10-11T12:40:47.791073601Z
	level=debug
	msg="Setting next state"
	handler=resultAlerting
logger=ngalert.state.manager
	rule_uid=edyspulkwxmgwc
	org_id=1
	instance=
	t=2024-10-11T12:40:47.791118337Z
	level=debug
	msg="Keeping state"
	state=Alerting
	previous_ends_at=2024-10-11T12:42:20Z
	next_ends_at=2024-10-11T12:42:40Z
logger=ngalert.image
	rule_uid=edyspulkwxmgwc
	org_id=1
	dashboard=cdxt4i23o5on4d
	panel=2
	t=2024-10-11T12:40:47.791172701Z
	level=debug
	msg="Requesting
	screenshot"
logger=ngalert.state.manager.persist
	t=2024-10-11T12:40:47.791229333Z
	level=debug
	msg="Saving alert states"
	count=1
	max_state_save_concurrency=1
logger=ngalert.state.manager.persist
	t=2024-10-11T12:40:47.806882121Z
	level=debug
	msg="Saving alert states done"
	count=1
	max_state_save_concurrency=1
	duration=15.651479ms
logger=secrets.kvstore
	t=2024-10-11T12:40:48.891954891Z
	level=debug
	msg="got secret value from cache"
	orgId=1
	type=datasource
	namespace=prometheus