Search with Keyword

Grafana Loki
1

How do I list all the pin_err_** codes using query in Grafana loki?
when a query unable to grouo it by error code.
any help is appreciated

2

Some example log would be helpful.

3

2023-03-06 10:38:34
4a98ddd4f690eaa9839e268483ca4461db232fb046a63d0046b02eab6cbcceee
flush_thread_1
kt999cc15w005
pod-754d556d56-wp4pt
ff019aa7-02b6-404b-a495-3eb4d0a57f49
{“log”:“E Mon Mar 6 10:38:34 2023 pod pod:1344396 pod_utils.c(155):6405 2:wsm-tomcat-64f47d9678-b59jh:WebService3715:1678095514:15674::::\n\n\tpod pod->FM input flist: opcode=Ppod_OP_ECE_POST_COMMIT, flags=0x0, errno=PIN_ERR_UNKNOWN_EXCEPTION:66\n\n# number of field entries allocated 20, used 1\n\n0 PIN_FLD_POID POID [0] 0.0.9.8 /publish 0 0\n”,“stream”:“stderr”,“time”:“2023-03-06T09:38:34.764133053Z”}

2023-03-06 10:38:34
4a98ddd4f690eaa9839e268483ca4461db232fb046a63d0046b02eab6cbcceee
flush_thread_1
kt999cc15w005
pod-754d556d56-wp4pt
ff019aa7-02b6-404b-a495-3eb4d0a57f49
{“log”:“E Mon Mar 6 10:37:01 2023 pod pod:1344150 fm_utils_publish.c:755 2:wsm-tomcat-64f47d9678-b59jh:WebServic3715:1678095420:22719::::\n\n\tPpod_OP_ECE_POST_COMMIT returned an error\n\n\t<location=PIN_ERRLOC_EM:18 class=UNKNOWN:0 errno=PIN_ERR_UNKNOWN_EXCEPTION:66>\n\n\t<field num=0:0,0 recid=0 reserved=0 reserved2=0 time(sec:usec)=1678095:421275>\n\n\t<facility=0 msg_id=0 version=0>\n\n341918165 32 /oms/virtual_time/shared/pin_virtual_time_file\n”,“stream”:“stderr”,“time”:“2023-03-06T09:37:01.276536541Z”}
2023-03-06 10:38:29
dc2f8ff9af063e946579be080dabf2b42ceb25269b8447bc97e51df6d9235d7a
flush_thread_5
kt999cc15w004
pod-754d556d56-z7lh4
f2203d34-8a30-4651-9243-fa97987600bb
{“log”:“E Mon Mar 6 10:38:29 2023 pod pod:1339177 pod_utils.c(155):6405 2:wsm-tomcat-64f47d9678-b59jh:WebService3715:1678095508:15633::::\n\n\tpod pod->FM input flist: opcode=Ppod_OP_ECE_POST_COMMIT, flags=0x0, errno=PIN_ERR_UNKNOWN_EXCEPTION:66\n\n# number of field entries allocated 20, used 1\n\n0 PIN_FLD_POID POID [0] 0.0.9.8 /publish 0 0\n”,“stream”:“stderr”,“time”:“2023-03-06T09:38:29.349802685Z”}

4

Your step should be:

  1. Parse with JSON.
  2. Extract the log field.
  3. Parse with logfmt or pattern.
  4. Extract the errno field.
  5. Aggregate.

Example:

sum by (errno) (
  count_over_time(
    {your_label=your_label_value}
      | JSON
      | line_format "{{.log}}"
      | logfmt
    [5m]
  )
)

If logfmt doean’t work, write your own pattern to match with.

5

Thanks
But something Im missing here what it can be-

sum by(ERRNO) (rate({namespace=~“abc-app”, container=~“xxx”} |= PIN_ERR_ | json | line_format {{.log}} | logfmt [5m]))

6

What do you get if you simply run your query without aggregation? Like what does this give you?

{namespace=~“abc-app”, container=~“xxx”} |= PIN_ERR_ | json | line_format {{.log}} | logfmt
7

I get good formatted log string
something like below-

What Im looking for is just errno= PIN_ERR_xxxx to be summed
i tried adding label filter but it would give me only one error number and I dont want to hard code any error code

Blockquote
“abc-app”, container=~“xxx”} |= PIN_ERR_ | json | line_format {{.log}} | logfmt | errno=PIN_ERR_BAD_VALUE:46>

Log labels

E

Mar

ccc_OP_BILL_MAKE_BILL

Wed

_05_00_22

_2023

2_pje_pin_bill_accts_2279318__437257408_110_1678248010_60199_root_0_0_0_1__

_8

_facility 0

_field

_location PIN_ERRLOC_CM:3

an

application oc-xx-helm-chart

class PIN_ERRCLASS_SYSTEM_INDETERMINATE:2

cm

cm_1627019

cm_child_c_141__5131

component xx-yyy

container xxx

container_id xxxxxxxd9235xxx

env pcrf15-t

errno PIN_ERR_BAD_VALUE:46>

error

fluentd_thread flush_thread_3

host anvcnbcbc

8

Not entirely sure what you are looking for. Are you looking for a specific error number? Or are you looking to just extract the number only?

9

I’m looking to find all the unique error codes and then show the count of their occurrence in chart

Like pin_err_ bad_value count 47
Pin_err_ xxx count 48
Etc etc

10

any other great ideas to unblock me?