Timestamp hh:mm:ss

[federicos1]

I’m setting up Grafana/Loki for the first time and I’m having trouble to set up the label for timestamp. I’m using this config.yml for Promtail:

scrape_configs:
- job_name: system
  pipeline_stages: 
    - regex: 
        expression: '^(?P<time>\d{2,2}:\d{2,2}:\d{2,2}).{4}(?P<level>(INFO)).*(?P<cpu>(\d{2,2}.\d{2,2}))\%.*(?P<ram>\s\d*.\d{2,2}).Mb+$'
    - labels: 
        cpu:
        level:
        ram:
    - timestamp: 
        format: "15:04:05"
        source: time
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

Here’s a sample for the log:

10:24:46.83 INFO [System Manager] CPU: 47.22%. Used RAM: 2251.01 Mb

Already had changed the limits_config in Loki:

limits_config:
  reject_old_samples: false

My question is… Does Promtail support this kind of timestamp format? Is there anyway to adapt this kind of format to override the final time value of the log that is stored?

[karsten]

Hi,

it seems you are missing the fraction of the seconds timestamp | Grafana Labs "15:04:05.000". Maybe that works.

[federicos1]

Hi karsten,

I didn’t include it on porpuse because of the format in the log that use two decimals instead of three. Nevertheless, I tried without changing the reg expression and I’m receiving no labels:

[karsten]

Sorry for my dumb question but did you set the proper time range in Grafana to look back? Also, whati sthe output of promtail’s server /metrics? What’s the value for promtail_sent_entries_total?

[federicos1]

My bad! I tried again setting the time range for the last 7 days and now I can see data. Despite that, I still can’t browse all the labels I configured:

I entered http://localhost:3100/metrics but can’t find any promtail_sent_entries_total. Should I paste the hole output for better understanding?

[karsten]

Hi, a colleague just suggested to use Troubleshooting | Grafana Labs to figure out where the labels are lost.

If you see data it should be alright. Also, I don’t think you should add CPU and RAM usage as labels for the stream. This would create a new log stream per label value. You should rather create the labels when you query the logs. I know this is a little confusing.

[federicos1]

Thank you @karsten !!! I’ll take your advice and leave for later the labels. Instead I’m going to focus in the time value and how to manipulate it. Since it is a log that is extracted on demand, it shouldn’t be difficult to format it to timestamp before promtail processes it.

[karsten]

    - timestamp: 
        format: "15:04:05.000"
        source: time

is not working?

[federicos1]

Not working. I also tried by removing the milliseconds decimals in the reg expression, but nothing. So I figured out that it is easier to add the “YYYY-MM-DD” as a prefix for each line (the log covers the events of one day) since the it’s a “estatic” log file. Once it’s extracted, it doesn’t change again.
The objective is to create some kind of analysis based on the metrics of the log, like CPU usage. That’s why I created a group CPU as label in the stream. But again, it’s my first time working with Loki/Promtail!

[karsten]

Hm, maybe the regex is not working then. Could you try ^(?P<time>\d{2,2}:\d{2,2}:\d{2,2}.\d{1,3}).*$? That only matches the time. The level would be a good label as well but let’s build it up.

[federicos1]

I aplied that expresion, leaving the config.yml like this:

scrape_configs:
- job_name: system
  pipeline_stages: 
    - regex: 
        expression: '^(?P<time>\d{2,2}:\d{2,2}:\d{2,2}.\d{1,3}).*$'
    - timestamp: 
        format: "15:04:05.000"
        source: time
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

Then I restarted the services and generated a panel, but it’s still the same :[

[karsten]

It might not process the logs if they’ve been processed. You have to clear the positions file Configuration | Grafana Labs.

[federicos1]

Ok! I cleared the file, restarted the services and then the same

[karsten]

Alright. I had to ask in the team. The match should be "15:04:05.999". However, you don’t have a date thus you get the current date.

[federicos1]

Hey! Turns out that the problem was the lack of date data so a entered (for now) manually in the log and now I can pass the time Label with no problems!!

Now, I have a next problem: Graph CPU asuge over time. I managed to reach this query:

But how can use the time/CPU data to graph it?

[karsten]

Nice! That’s what I was hoping in my last comment

If I’m not mistaken you need to unwrap the label:

{job="varlogs"} | regex "^.(?P<CPU>(...))" | unwrap CPU

Maybe put a rate around it

rate({job="varlogs"} | regex "^.(?P<CPU>(...))" | unwrap CPU [10m])

Checkout the docs on Metric Queries for more details.