Why would Grafana / Loki give inconsistent results?

Grafana Loki
,
5

Thank you this is information I’ve been wondering about.

I tried to create a directory to mount into the loki container, but it was having permission issues. I was creating the directory as root but the /loki folder in the container is owned by loki user. Similar permissions issue w/positions.yaml but I think I found out how to resolve that here ( Promtail fail to write positions.yaml file mounted · Issue #8543 · grafana/loki · GitHub). So, I ended up creating a named volume to map to the /loki folder and I created a local directory to map as the /tmp in promtail’s container. Not sure if this is the all that needs to be done, but I copied my new docker-compose.yaml below in case it helps anyone in the future.

I have tried this as well, but was having the same issues described here ( Timestamp isn't working for NGINX logs - #2 by armesh49 ). After loading the data with the custom timestamp I wouldn’t be able to get proper query results for hours. That forced me to stop using the custom timestamp to be able to speed up troubleshooting the other issues.

I am fine with restarting everything from scratch with empty logs and a clean install of this setup once I can figure out how to get consistent behavior out of these dashboards. At this point I’m pretty sure it’s Loki giving the inconsistent results. If it’s showing up in my Grafana Dashboards then Loki must be getting the data properly from Promtail. My next guess was to try to query loki for all of its entries and start to see if the issue was with Grafana or Loki. Perhaps mounting these volumes fill affect things. I’ll wait a bit and see, but still open to ideas on how else to troubleshoot.

docker-compose.yaml 
version: "3"

networks:
  loki:

volumes:
  loki-main:

services:
  loki:
    container_name: loki
    image: grafana/loki:2.9.3
    restart: unless-stopped
    ports:
      - "3100:3100"
    volumes:
      - /var/lib/docker/volumes/loki/etc/loki/local-config.yaml:/etc/loki/local-config.yaml
      - loki-main:/loki
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  promtail:
    container_name: promtail
    image: grafana/promtail:2.9.3
    restart: unless-stopped
    volumes:
      - /var/www/vhosts:/var/www/vhosts
      - /var/lib/docker/volumes/promtail/etc/promtail/config.yml:/etc/promtail/config.yml
      - ${PWD}/promtail-tmp:/tmp
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

  grafana:
    container_name: grafana
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
        - name: Loki
          type: loki
          access: proxy
          orgId: 1
          url: http://loki:3100
          basicAuth: false
          isDefault: true
          version: 1
          editable: false
        EOF
        /run.sh
    image: grafana/grafana:latest
    restart: unless-stopped
    ports:
      - "3000:3000"
    networks:
      - loki