Loki: cisco logs trimmed when shown in Grafana

[ralvez]

Hello team.
I am looking for guidance on the following matter.

The scenario:
I was asked to mount a solution to receive/show SYSLOG messages coming from Cisco devices.
Since I already have Grafana, I installed a Loki docker, and then I mounted a “syslog-ng” docker to receive the logs on port 514 and forward them to promtail´s docker on port 1514 (AFAIK promtail does not hear messages on UDP 514).

The problem:
I configured a Cisco router to include a string representing its hostname when logs are sent. The following is a sample of the message captured with my sniffer. “RO-AR-PL-PUA-001” is the string I configured in the router to be sent along the log message (command: logging origin-id string RO-AR-PL-PUA-001)

340: RO-AR-PL-PUA-001: 000336: *Dec 14 08:57:03.629: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.235.254.3 port 514 stopped - CLI initiated

When I ask Grafana to show me the log, the aforementioned message is trimmed. I see the following:

000336: *Dec 14 08:57:03.629: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.235.254.3 port 514 stopped - CLI initiated

I do not know which of my “syslog dockers” (syslog-ng or promtail) is getting rid of the first piece of the log message. I need it to work with this information for the purpose of a label associated to the name of the device.

¿ How could I start debugging the problem?
Any hints will be greatly appreciated

Best regards,
Rogelio

[tonyswumac]

Although I haven’t had to use promtail for syslog, it does look like you can configure promtail to listen to any port you want with syslog. See Configure Promtail | Grafana Loki documentation. It only supports RFC5424 though.

Also as you can see in the documentation, there are many labels / metadata that comes with syslog that aren’t part of the logline. I’d say try to get rid of your syslog docker and let promtail listen to syslog directly, and then try to set some of those labels and see what they look like in Loki.

[ralvez]

Hi Tony:
I am forced to use "something " between the Cisco devices (RFC3164, UDP port 514) and promtail (RFC5424). In fact, the promtail documentation pointed by you does mention this need:

…The syslog block configures a syslog listener allowing users to push logs to Promtail with the syslog protocol. Currently supported is [IETF Syslog (RFC5424)]

The recommended deployment is to have a dedicated syslog forwarder like syslog-ng or rsyslog in front of Promtail. The forwarder can take care of the various specifications and transports that exist (UDP, BSD syslog, …

Regarding labels: I played a bit with them via regex trying to fetch the “hostname” information, but I fear the message was already trimmed.

¿ Does promtail by default store the logs in some file in addition to sending them to Loki?

Rogelio

[tonyswumac]

You might try to configure your syslog container to log to a file or console then, and see what it looks like. And if the information you are looking for is already trimmed then you can use syslog template to append to the log lines.

[ralvez]

Hi Tony:
I am struggling with Syslog-ng but the documentation is scarce.

I will see whether I can setup Promtail to store the incoming messages in a file; there should be a way.

Best regards

[scottfingerhut]

OK. I honestly don’t know if this is it, but I went over to the Grafana site and asked the question in Grot (the AI bot). I’m not very technical, so I don’t know if this is good advice, but would love to hear if it’s decent or not.

From Grot:
To start debugging the problem, you can follow these steps:

1. Check the configuration of your syslog-ng docker container. Ensure that it is properly configured to receive logs on port 514 and forward them to promtail’s docker on port 1514. Verify that there are no filters or transformations in syslog-ng that might be removing or modifying the log message.

2. Verify the configuration of your promtail docker container. Make sure that it is correctly configured to receive logs from syslog-ng on port 1514. Check if there are any relabeling or filtering rules in promtail that might be affecting the log message.

3. Enable debug logging in both syslog-ng and promtail containers. This will provide more detailed information about the processing of log messages. You can check the logs of both containers to see if there are any errors or warnings related to the log message trimming issue.

4. Capture network traffic between the Cisco router and the syslog-ng container using a packet sniffer. This will allow you to inspect the actual log messages being sent by the router and received by syslog-ng. Verify if the hostname string “RO-AR-PL-PUA-001” is present in the log message at this stage.

5. If the hostname string is present in the log message received by syslog-ng, but missing in the log message processed by promtail, then the issue might be with promtail’s configuration or processing. Review the relabeling rules in promtail’s configuration file to ensure that the hostname label is correctly extracted from the log message.

6. If the hostname string is missing in the log message received by syslog-ng, then the issue might be with syslog-ng’s configuration or the way the Cisco router is sending the logs. Double-check the syslog-ng configuration and verify if the router is correctly configured to include the hostname string in the log messages.

By following these steps, you should be able to identify the source of the problem and determine whether it lies in the syslog-ng or promtail configuration.

[ralvez]

Hello Scott:
Thank you very much!

Please have in mind that I do see the logs in Grafana, so the path router > Syslog-nd > Promtail > Loki > Grafana is OK. The only real issue here is that only a piece of the information in each log is missing when rendered in Grafana.

I managed Syslog-nd to write the log messages to a file in its docker. I modified its configuration file with the following sentence:

log {
source(s_local);
source(s_network);
destination(d_loki);
destination { file(“/var/log/syslog”); };
};

After that, I restarted the docker, and was able to see the following:

root@7a99e08af45b:/var/log# cat syslog
Dec 14 20:01:40 10.133.48.10 RO-AR-PL-PUA-001: 000433: *Dec 14 20:04:50.487: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback667, changed state to down
root@7a99e08af45b:/var/log#

So the missing string “RO-AR-PL-PUA-001” is still there in SYSLOG-ND.

I will assume that the same string is sent to Promtail. Under this assumption, I will start working in Promtails´ docker to see why that piece of the string is missing.

I will keep you posted.
Thanks!

[ralvez]

@ralvez

Dec 14 20:01:40 10.133.48.10 RO-AR-PL-PUA-001: 000433: *Dec 14 20:04:50.487: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback667, changed state to down

Hello Tony:
I was able to make SYSLOG-NG write to a local file. Now I have something to work with.
I will keep you posted.
Thanks!

[ralvez]

Guys:
I wonder if promtail can push the logs to a local file in addition to Loki.

The piece of related content in promtail´s YAML configuration file is the following:

clients:

• url: http://loki:3100/loki/api/v1/push

¿ Can i add another line of some sort like “file: /var/log/debug.txt”?

[tonyswumac]

1. I don’t think promtail can output to a file. at least i don’t know how to. There is a way to run promtail with inspect turned on (see Troubleshooting Promtail | Grafana Loki documentation). You can try this and see if anything shows up in the console. I’ve not done this before with a live listener, so I don’t know how well it would work.

2. I think the fields you don’t see in promtail are probably being parsed as labels. See Configure Promtail | Grafana Loki documentation for a list. I’d say configure all of them in promtail, and see what shows up in Grafana.

[ralvez]

Hello Tony:
I have been doing a lot of tests with mixed results.

Bottom line I was unable to expose new labels, at least in Grafana. Given for example the following log:

*Dec 15 18:30:14 10.133.48.10 XX hostname=RO-AR-PL-PUA-001 devicetype=L2 country=ARG site=ARG_PL_PUA : 000625: Dec 15 18:33:27.113: %SYS-5-CONFIG_I: Configured from console by ralvez on vty0 (10.235.254.2)

Grafana offers me the following labels to choose from:

I am pretty sure that the aforementioned labels appear in Grafana because they are created in Promtail´s YAML file:

The commented lines in this piece of the current YAML file belong to my attempt to make promtail create new labels by parsing the log message.
As it can be seen, I am trying to generate a label called “devtype” if I match “devicetype=L2” in the text of the syslog message. The sample message has this string in it, but Grafana does not seem to detect it.

By other hand: I found that if I “explore” the logs, and I include the “logfmt” function in the interactive query, then Grafana starts to render a lot of “labels”:

I attempted to include the “logfmt” tool inside promtail´s YAML file. Since the previous image showed me that “logfmt” exposed some fields of the message as labels in Grafana, then I supposed that this tool in the YAML file might have exposed new labels when promtail was pushing the logs to Loki.
The bad news here is that I was unable to find a way to use “logfmt” in the YAML file: every time I included it, promtail was not able to come up. I did not know how to insert the “logfmt” stanzas in the YAML file. ¿ inside the syslog section? ¿globally? ¿with or without a hyppen character?

Sorry for the long message. Have a nice week end.
Rogelio

[tonyswumac]

There is no logfmt in promtail, but you can still extract strings as labels. Most of the time you’d use regex group capture, then set labels afterwards. Let’s say given log string:

Dec 15 18:30:14 10.133.48.10 XX hostname=RO-AR-PL-PUA-001 devicetype=L2 country=ARG site=ARG_PL_PUA : 000625: Dec 15 18:33:27.113: %SYS-5-CONFIG_I: Configured from console by ralvez on vty0 (10.235.254.2)

Which part do you want to extract as labels? I may be able to give you an example configuration.

[ralvez]

Hello Tony:
I could have been struggling with logfmt all day long without any result whatsoever!!!

Regarding your question: in the sample message I would like, for example, to generate a new label called “devtype” if I find the “devicetype” string in the message. Then I want to assign to it the string after “devicetype=”; in this case: “L2”

In order to achieve the aforementioned, I attempted the following sentences without success:

The previous regex, in my humble opinion, should capture as wished:

1. first I use “.*” to skip 0 or more characters until I find the following
2. then I look for the “devicetype=” string
   3)I include a capture object with name “devtype” that looks for 2 (two) chars in the A-Z0-9 group
3. then I included .* to include 0 or more characters after the previous

[ralvez]

@tonyswumac

There is no logfmt in promtail

Hello Tony:
¿ Might the documentation be wrong? I attempted logfmt in the YAML file since in the following URL I found it should work:

logfmt | Grafana Loki documentation

[ralvez]

Guys: I made some progress.

The problem is related to the nature and/or structure of Cisco´s logs. They are not really SYSLOG messages according to RFC5424.

A syslog message should have the following format:
date, time, host name, a username and the text of the log message itself.

A Cisco router (at least on traditional IOS; Cisco has many different operating systems depending on the product line) generates this kind of log:

Dec 15 16:44:00 000617: *Dec 15 16:47:13.805: %SYS-6-TTY_EXPIRE_TIMER: (exec timer expired, tty 866 (10.235.254.2)), user ralvez

The date appears twice (the 2nd date and time, in the body of the message, should not be there), there is no hostname or IP and instead you get a 6 digit number, and so on.

Promtail, for instance, expects a standard syslog message, so it gets the 6 digit number of the log and associates this number with the hostname, as can be seen in Grafana:

I made some research in the syslog-ng community. This team is already aware of the problem with Cisco logs, and have provided a “Cisco parser” that can process a Cisco log in order to deal with (some or perhaps all of) these issues. The following is a Cisco log that has been “standardized” by syslog-ng’s parser; the “hostname” still is 000691, but the 2nd date was erased from the original message.

Dec 16 09:45:38 000691 %SYS-6-LOGOUT: User ralvez has exited tty session 866(10.235.254.2)

Now I proceeded to edit promtail´s YAML file to give it a try with logs like the previous one, searching for the "User " string and looking to generate a “UserID” label:

Result: Grafana exposed the label!!!

I will have to decide how to approach the subject.

Just to let you know my progress on the matter. Keep posted :o)

Thanks, Rogelio

PS: the following syslog-ng document was really useful to start working the problem out

Parsing Cisco logs in syslog-ng

PS2: you can configure the Cisco router with the “loggin origin-id …” command to add some information like the router´s IP or hostname, but this is not necessarily the solution to the problem. In fact, the log message still remains odd for promtail. Nonetheless, now I have much more information and context to approach the solution.

[ralvez]

Guys, much better.

I added a fake “userID” to the Cisco logs generated by the router, using a “XX” string prepended to the log. Basically, i added the following configuration to the router:

logging origin-id string "XX hostname=RO-AR-PL-PUA-001 devicetype=L2 country=ARG site=ARG_PL_PUA "

Now the log looks like this in syslog-ng: I have a date, a time, a hostname and a fake XX username. The rest of the message is the actual message payload, and now I can work with it in promtail:

*Dec 16 10:59:52 10.133.48.10 XX hostname=RO-AR-PL-PUA-001 devicetype=L2 country=ARG site=ARG_PL_PUA : 000697: Dec 16 11:03:07.142: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: ralvez] [Source: 10.235.254.3] [localport: 22] at 11:03:07 ARG Sat Dec 16 2023

I modified again my promtail YAML with the objective to generate a “hostID” label if I find “hostname=” in the message:

See the label in Grafana

I will have to take a lot of care because logs coming from different network devices can be different between them. But I have gained a lot of insight on the problem.

Thanks everybody! Thanks Tony!

l

[ralvez]

Done!

Modified promtail YAML file…

Grafana:

[tonyswumac]

Glad you were able to figure it out! It’s definitely good information to know that Cisco does their own thing (as they often do).

[jkpparker]

Do you mind sharing your whole promtail config, tyring something similar, but cant get it to work

[ralvez]

Sure no problem.

Please have in mind that the solution involves proper configuration in

• the cisco devices: I had to build the “logging origin-id string” command with a dummy prefix to make it work according to what it is expected from a SYSLOG message.
• syslog-ng: it may need to include the “cisco-parser()” sentence depending on how you want to pass your SYSLOG messages to Promtail
• promtail itself.

Besides the aforementioned, I found that different IOS versions on the same hardware do not send the logs with the same structure. If you prepare your sentences waiting for some specific pattern, it may not work with one IOS and may work in the next available IOS. I found, for instance, that in Cisco 9200-24P switches the log will not be processed by my docker if the switch is running 17.3.4 whereas it works smoothly in 17.3.5

For the record: besides the log itself, I want my Cisco devices to notify additional information such us site, country, device type, device model, IOS version, and so on. The following is how I do it in a router:

logging origin-id string “XX hostname=RO-AR-PL-PUA-001 devicetype=RO country=AR site=AR-PL-PUA vendor=CISCO model=ISR4351 version=16.9.6”

The “XX” at the very beginning of the string is needed because SYSLOG messages are expected to have the following structure: DATE TIME HOSTNAME USERNAME LOGMSG. But Cisco does not obey this model; a log message is usually preceded by DATE, TIME and a word that depending on the platform does not represent the hostname, nothing more. So I added a dummy “XX” at the very beggining of the string to build the 4th parameter in the prefix of the log message itself.

The “vendor=CISCO” in my string is because I want to use it in promtail to do something if the log comes from a CISCO device. I am preparing my log system for the future addition of the devices from other vendors (Fortinet, PaloAlto, …)

The following is my current Promtail YAML. Let me know if you need more explanation. Best regards

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:1514
      idle_timeout: 60s
      label_structured_data: no
      labels:
        job: "syslog"
    relabel_configs:
      - source_labels: ['__syslog_message_hostname']
        target_label: 'host'
    pipeline_stages:
      - match:
          selector: '{job="syslog"} |= "CISCO"'
          action: keep
          stages:
            - regex:
                expression: '.*hostname=(?P<hostname>[A-Z0-9_-]{16}).*devicetype=(?P<devicetype>[A-Z0-3]{2}).*country=(?P<country>[A-Z]{2}).*site=(?P<site>[A-Z_-]{9}).*vendor=(?P<vendor>[A-Z]+).*model=(?P<model>[A-Z0-9_-]+).*version=(?P<version>[a-zA-Z0-9().]+).*[%]([A-Z0-9\_])+[-](?P<loglevel>[0-7]).*'
            - labels:
                hostname:
                devicetype:
                country:
                site:
                vendor:
                model:
                version:
                loglevel:
      - match:
          selector: '{job="syslog"} != "CISCO"'
          action: drop