Query pods with error state (not cronjobs)

Hi,
I got a question about Promql (without using Grafana).
I am using the following query to obtain all the pods that have Error status

count(kube_pod_container_status_terminated_reason{reason='Error'} > 0) by (namespace, pod)

The issue is that this results in both pods and cronjobs being checked.
Is there a way to narrow down the metric to be only for pods or only for cronjobs?

I read online that cronjobs have label: job-name but i am unsure how to utilize this information

Thank you

Hi,
Can you provide list of labels you have in this metric? I think that kubernetes reports this metric and doesn’t really care if the pod is a service or a cronjob - a pod is a pod but maybe you have a label in there that would show something? If not, you can try to change the scrape-config to include such label based on some k8s annotations / labels. (Scrape config wouldn’t help either, since it’s a label from the k8s api pod / kube-metrics or something).

I read online that cronjobs have label: job-name

I guess it refers to pod labels which are not passed by default to metrics (you’d have to check in your scrape-config if it would be possible to add the label).

You could theoretically find a metric with pod label that is reported only from services and not cronjobs (if you have something like this) and join - it would drop all cronjobs pod_container_status.

Thank you for reply, the metric used comes from kube-state-metric which is native solution. Here is a link and below I add labels:

container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
reason=<container-terminated-reason>
uid=<pod-uid>

I was hoping to find maybe a combination of queries that could result in filtering the pods vs cronjobs. The kube_pod_container_status_terminated_reason natively doesnt support such filtering but maybe there is another one that would work.
I will keep looking, maybe in the meantime someone who tried to solve this issue will give advice :slight_smile:

Ok, I found a metric kube_cronjob_created (which is here, so tell me if you have it or not). If you have it, you can create an unless expression which would say “give me the series from the left side unless there’s the same series on the right side”. So the idea is we know the left side contains cronjobs and services and the right side contains only cronjobs - the difference of sets will be services.
Left side:
label_replace(count(kube_pod_container_status_terminated_reason{reason='Error'} > 0) by (namespace, pod), "cronjob", "$1", "pod", "(.*)-.*-.*") - to the query you have created we’re adding cronjob label (theoretically we could use already present container but it doesn’t always have to be the same I guess). Also we named it cronjob because we can’t join on different labels - they have to have the same name.

Right side
count(kube_cronjob_created{}) by (cronjob)

Result query:
label_replace(count(kube_pod_container_status_terminated_reason{reason='Error'} > 0) by (namespace, pod), "cronjob", "$1", "pod", "(.*)-.*-.*") unless on(cronjob) count(kube_cronjob_created{}) by (cronjob)

In my mind (and from my tests) it should work, but I don’t have any services in error state. Hope that helps!

Edit: alternatively you can use kube_replicaset_created or kube_deployment_created to join on services (you’d have to check if there are only services, but I don’t think cronjob creates deployments / replicasets). Then you’d use and instead of unless (the rest is the same)

Thanks a lot for taking the time to look deeper into it.
The query provided doesn’t work for me, and I am confident I know why.
I think to solve my problem what I need is the ability to define what labels in pod definition I want to filter on, and forget about the cronjob approach.

If I can find a why to combine the query:
count(kube_pod_container_status_terminated_reason{reason='Error'} > 0) by (namespace, pod) with the ability to sort by pod label that should work

Can you share what doesn’t work with the query? Also - which “labels in pod definition” do you mean? The labels from Pod (e.g. obtained by kubectl describe pod) probably won’t be in your metrics by default.

Sorry for confusion I made some more investigation and found out that I had to enable kube_pod_labels metric in my helm chart.

For those looking at this in the future, add this to helm chart (if using kube-prometheus-stack)

kube-state-metrics:
  metricLabelsAllowlist:
    - pods=[job-name]

I have done that and now I got the following query
count(kube_pod_status_reason{reason='Evicted', cluster='test', namespace='brain'} > 0) by (namespace, pod)
This gives me all the pods (pods + cronjobs) in given cluster + namespace that have been evicted.
Now I wish to filter out the pods which do not have label=job_name present.
I then do: kube_pod_labels{cluster='toolset', namespace='brain'} and as a result I can see the following:

From this image its clear that cronjobs have label called label_job_name and pods do not have it.

So having said that, I reformulate my question :slight_smile:
How can I query evicted pods but specify that I want just pods or just cronjobs.

Ok, so:

Filtering only cronjobs / only non-cronjobs pods from kube_pod_labels metric

You can do that by specifying whether the label label_job_name should exist.
Query kube_pod_label{label_job_name=""} will return all the series that do not have the label label_job_name and query kube_pod_label{label_job_name=~".+"} will return only series that have the label_job_name set to anything (.+ is important because it does not match an empty string like .* does). - PromQL assumes that empty string equals “label doesn’t exist”

Connecting the result to kube_pod_status_reason

You still have to join the two queries. I still think the query from my response should work but now you also have pod label, so it could look like this:
count(kube_pod_status_reason{reason='Evicted', cluster='test', namespace='brain'} > 0) by (namespace, pod) and on(pod) kube_pod_label{cluster='test', namespace='brain'}.

Also I don’t think grouping by namespace is needed, since in the query you provided the namespace - it will always be the same.

1 Like

Dzękuję bardzo za twoją pomoc :slight_smile:
(Thanks for your help)

Here are the final queries that worked:

Cronjobs
count(kube_pod_status_reason{reason='Evicted', cluster='test', namespace='brain'} > 0) by (pod) and on(pod) kube_pod_labels{label_job_name!=""}

Pods
count(kube_pod_status_reason{reason='Evicted', cluster='test', namespace='brain'} > 0) by (pod) and on(pod) kube_pod_labels{label_job_name=""}

Really appreciate your help!