# Why would Grafana / Loki give inconsistent results?

**URL:** <https://community.grafana.com/t/why-would-grafana-loki-give-inconsistent-results/110984>\
**Category:** Grafana Loki\
**Tags:** loki, grafana\
**Created:** [December 30, 2023, 11:29pm UTC](https://community.grafana.com/t/why-would-grafana-loki-give-inconsistent-results/110984 "2023-12-30T23:29:21Z")\
**Posts on this page:** 1\
**Showing post:** 5

<div class="post-metadata">

**Author:** ![brianvb](https://avatars.discourse-cdn.com/v4/letter/b/b4bc9f/32.png) [@brianvb](https://community.grafana.com/u/brianvb)\
**Post date:** [January 4, 2024, 4:15am UTC](https://community.grafana.com/t/why-would-grafana-loki-give-inconsistent-results/110984/5 "2024-01-04T04:15:21Z")

</div>

Thank you this is information I’ve been wondering about.

I tried to create a directory to mount into the loki container, but it was having permission issues. I was creating the directory as root but the /loki folder in the container is owned by loki user. Similar permissions issue w/positions.yaml but I think I found out how to resolve that here ( [Promtail fail to write positions.yaml file mounted · Issue #8543 · grafana/loki · GitHub](https://github.com/grafana/loki/issues/8543)). So, I ended up creating a named volume to map to the /loki folder and I created a local directory to map as the /tmp in promtail’s container. Not sure if this is the all that needs to be done, but I copied my new docker-compose.yaml below in case it helps anyone in the future.

> [@tonyswumac](#):
>
> But the best way to fix this is to parse for timestamp in promtail so that logs that are sent to Loki uses the timestamp specified in the logs themselves rather than the time of which when promtail attempts to collect and send logs.

I have tried this as well, but was having the same issues described here ( [Timestamp isn't working for NGINX logs - #2 by armesh49](https://community.grafana.com/t/timestamp-isnt-working-for-nginx-logs/75869/2) ). After loading the data with the custom timestamp I wouldn’t be able to get proper query results for hours. That forced me to stop using the custom timestamp to be able to speed up troubleshooting the other issues.

I am fine with restarting everything from scratch with empty logs and a clean install of this setup once I can figure out how to get consistent behavior out of these dashboards. At this point I’m pretty sure it’s Loki giving the inconsistent results. If it’s showing up in my Grafana Dashboards then Loki must be getting the data properly from Promtail. My next guess was to try to query loki for all of its entries and start to see if the issue was with Grafana or Loki. Perhaps mounting these volumes fill affect things. I’ll wait a bit and see, but still open to ideas on how else to troubleshoot.

> **docker-compose.yaml**
>
> ```auto
> version: "3"
> 
> networks:
> loki:
> 
> volumes:
> loki-main:
> 
> services:
> loki:
> container_name: loki
> image: grafana/loki:2.9.3
> restart: unless-stopped
> ports:
> - "3100:3100"
> volumes:
> - /var/lib/docker/volumes/loki/etc/loki/local-config.yaml:/etc/loki/local-config.yaml
> - loki-main:/loki
> command: -config.file=/etc/loki/local-config.yaml
> networks:
> - loki
> 
> promtail:
> container_name: promtail
> image: grafana/promtail:2.9.3
> restart: unless-stopped
> volumes:
> - /var/www/vhosts:/var/www/vhosts
> - /var/lib/docker/volumes/promtail/etc/promtail/config.yml:/etc/promtail/config.yml
> - ${PWD}/promtail-tmp:/tmp
> command: -config.file=/etc/promtail/config.yml
> networks:
> - loki
> 
> grafana:
> container_name: grafana
> environment:
> - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
> - GF_AUTH_ANONYMOUS_ENABLED=true
> - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
> entrypoint:
> - sh
> - -euc
> - |
> mkdir -p /etc/grafana/provisioning/datasources
> cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
> apiVersion: 1
> datasources:
> - name: Loki
> type: loki
> access: proxy
> orgId: 1
> url: http://loki:3100
> basicAuth: false
> isDefault: true
> version: 1
> editable: false
> EOF
> /run.sh
> image: grafana/grafana:latest
> restart: unless-stopped
> ports:
> - "3000:3000"
> networks:
> - loki
> 
> ```

---

_[View the full topic](https://community.grafana.com/t/why-would-grafana-loki-give-inconsistent-results/110984)._