Need help creating Log Panel because it is always empty

Grafana Elasticsearch
,
1

I successfully created my first Grafana Dashboard, with a panel showing a basic count vs. time graph of log results from Elasticsearch.

I tried to add a Logs Panel but it is always empty. I even tried to duplicate the panel showing the log data (the graph panel), then change the Visualization to Logs.

When I go to Explore, I can see the log content when I change the Metrics/Log tab to logs. But I cannot reproduce this display on my Dashboard.

I am new to Grafana as well as Elasticsearch. Any help would be appreciated.

  • What Grafana version and what operating system are you using?

  • What are you trying to achieve?

  • How are you trying to achieve it?

  • What happened?

  • What did you expect to happen?

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

  • Did you follow any online instructions? If so, what is the URL?

2

please share obfuscated raw data sample from your elastic search?

3

Sure, here’s one example result row, with desired log message being

2022/05/09 14:39:55 [debug] 28#28: *242912 proxy.lua:26: new(): connection to myadmin.3scale.net:443 established, reused times: 1

Sample result row from Elasticsearch:

{
  "@timestamp": "2022-05-09T14:39:55.469Z",
  "message": "2022/05/09 14:39:55 [debug] 28#28: *242912 proxy.lua:26: new(): connection to myadmin.3scale.net:443 established, reused times: 1",
  "event.created": "2022-05-09T14:39:55.469Z",
  "event.kind": "event",
  "user.id": "0",
  "user.group.id": "0",
  "host.architecture": "x86_64",
  "host.hostname": "xxx",
  "host.id": "xxx",
  "host.name": "xxx",
  "host.os.name": "Red Hat Enterprise Linux Server",
  "host.os.kernel": "3.10.0-1160.59.1.el7.x86_64",
  "host.os.codename": "Maipo",
  "host.os.type": "linux",
  "host.os.platform": "rhel",
  "host.os.version": "7.9 (Maipo)",
  "host.os.family": "redhat",
  "host.containerized": false,
  "host.ip": [
    xxx
  ],
  "host.mac": [
    xxx
  ],
  "journald.gid": 0,
  "journald.host.boot_id": "xxx",
  "journald.pid": 1326,
  "journald.process.capabilities": "1fffffffff",
  "journald.process.executable": "/usr/bin/dockerd-current",
  "journald.process.command_line": "/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver=journald --signature-verification=false --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/vg00-docker--pool --storage-opt dm.use_deferred_removal=true --add-registry registry.access.redhat.com --add-registry registry.redhat.io --add-registry docker.io",
  "journald.process.name": "dockerd-current",
  "journald.uid": 0,
  "systemd.cgroup": "/system.slice/docker.service",
  "systemd.slice": "system.slice",
  "systemd.unit": "docker.service",
  "systemd.transport": "journal",
  "ecs.version": "8.0.0",
  "log.syslog.priority": 3,
  "container.id": "xxx",
  "container.name": "apicast",
  "container.log.tag": "xxx",
  "container.id_truncated": "xxx",
  "process.pid": xxx,
  "process.command_line": "/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver=journald --signature-verification=false --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/vg00-docker--pool --storage-opt dm.use_deferred_removal=true --add-registry registry.access.redhat.com --add-registry registry.redhat.io --add-registry docker.io",
  "process.args": [
    "/usr/bin/dockerd-current",
    "--add-runtime",
    "docker-runc=/usr/libexec/docker/docker-runc-current",
    "--default-runtime=docker-runc",
    "--authorization-plugin=rhel-push-plugin",
    "--exec-opt",
    "native.cgroupdriver=systemd",
    "--userland-proxy-path=/usr/libexec/docker/docker-proxy-current",
    "--init-path=/usr/libexec/docker/docker-init-current",
    "--seccomp-profile=/etc/docker/seccomp.json",
    "--selinux-enabled",
    "--log-driver=journald",
    "--signature-verification=false",
    "--storage-driver",
    "devicemapper",
    "--storage-opt",
    "dm.fs=xfs",
    "--storage-opt",
    "dm.thinpooldev=/dev/mapper/vg00-docker--pool",
    "--storage-opt",
    "dm.use_deferred_removal=true",
    "--add-registry",
    "registry.access.redhat.com",
    "--add-registry",
    "registry.redhat.io",
    "--add-registry",
    "docker.io"
  ],
  "process.args_count": 27,
  "syslog.priority": 3,
  "input.type": "journald",
  "agent.type": "filebeat",
  "agent.version": "8.1.3",
  "agent.ephemeral_id": "xxx",
  "agent.id": "xxx,
  "agent.name": "xxx"
}
4

not a valid json blob. please fix and post back?

5

It’s probably not valid because I manually edited the data to hide the real hostname, IP, etc.

Thanks for the help, anyway. I found a way to display what I needed in Kibana instead. I came here because we also have Grafana accounts and I thought maybe Grafana had what I needed and Kibana didn’t.

6

so let’s try it in grafana. I fixed the data issue. what kind of visualization do you want to show this data in and which portion of the data?