Slow query on larger logs

Grafana Loki
1

Hi,

I am new with Grafana Loki.

Queries run fine on logs that are only 3-5 lines per record. However, when I send in logs that are 20-30 lines per record, a query could take over a minute and then stopped.

If I do a Live tail, I can see logs coming in. But if I choose last 5m to display them, it takes forever.

What do you suggest that I should optimize in this setup? What value should I change?

Thank you.

1 Like
2

How are you deploying Loki? Can you share your configuration? Also have you checked Loki logs for any errors?

4

This is my config.


   storage_config:
     aws:
       region: us-west-2 # for example, eu-west-2
       bucketnames: xxxxxx # Your actual S3 bucket name, for example, loki-aws-dev-chunks
       s3forcepathstyle: false
   ingester:
       chunk_encoding: snappy
   pattern_ingester:
       enabled: true
   limits_config:
     allow_structured_metadata: true
     volume_enabled: true
     max_query_parallelism: 32
     retention_period: 672h # 28 days retention
   compactor:
     retention_enabled: true
     delete_request_store: s3
   ruler:
    enable_api: true
    storage:
      type: s3
      s3:
        region: us-west-2 # for example, eu-west-2
        bucketnames: loki-aws-logs-ruler # Your actual S3 bucket name, for example, loki-aws-dev-ruler
        s3forcepathstyle: false
      alertmanager_url: http://prom:9093 # The URL of the Alertmanager to send alerts (Prometheus, Mimir, etc.)

   querier:
      max_concurrent: 8

   storage:
      type: s3
      bucketNames:
        chunks: "xxxxxx" # Your actual S3 bucket name (loki-aws-dev-chunks)
        ruler: "xxxxxx" # Your actual S3 bucket name (loki-aws-dev-ruler)
        # admin: "<Insert s3 bucket name>" # Your actual S3 bucket name (loki-aws-dev-admin) - GEL customers only
      s3:
        region: us-west-2 # eu-west-2
        #insecure: false
      # s3forcepathstyle: false

serviceAccount:
 create: true
 annotations:
   "eks.amazonaws.com/role-arn": "arn:aws:iam::XXXXXXXXXX:role/LokiServiceAccountRole" # The service role you created

deploymentMode: Distributed

ingester:
 replicas: 3
 persistence:
   storageClass: gp3
   accessModes:
     - ReadWriteOnce
   size: 10Gi

querier:
 replicas: 3
 maxUnavailable: 2
 persistence:
   storageClass: gp3
   accessModes:
     - ReadWriteOnce
   size: 10Gi
queryFrontend:
 replicas: 2
 maxUnavailable: 1
queryScheduler:
 replicas: 2
distributor:
 replicas: 3
 maxUnavailable: 2
compactor:
 replicas: 1
 persistence:
   storageClass: gp3
   accessModes:
     - ReadWriteOnce
   size: 10Gi
indexGateway:
 replicas: 2
 maxUnavailable: 1
 persistence:
   storageClass: gp3
   accessModes:
     - ReadWriteOnce
   size: 10Gi
ruler:
 replicas: 1
 maxUnavailable: 1
 persistence:
   storageClass: gp3
   accessModes:
     - ReadWriteOnce
   size: 10Gi
5

Check in your query frontend logs and see if your query are being split. If not, you should consider properly configure query frontend for query splitting. See Query frontend example | Grafana Loki documentation.