Hi ,
Pretty new to Loki, just set up Loki 4 weeks ago and trying to test it…loaded our existing logs into Loki and when i try to run a query for time series of 7days the histogram says No logs Volume available( attaching screenshot for reference) for 7 days the query is trying to pull 150GB of data only…in real work per day per one app we push close to 1TB. Trying to see how i can get this histogram working for 1TB of data for 24hrs of time series. If anyone is aware, please share the setting or config that we need to tweak. Thanks in advance.
You have an empty filter. You should try and get rid of that.
I would also recommend you to use the Code view, I personally prefer that. And your query should just look like {application="ris20pos"}
Hi tony,
Firstly, thanks for your unwavering support. I still have the issue even after removing the empty filter.
I though it would be some setting on the reader or index gateway…how does the log volume histogram work? where does it fetch the data from?
Still no luck, if i change the time frame to last 2 days then may be it loads… When i search for a large time interval 1-2 weeks…the bar graph doesnt show up.
Check your Loki cluster logs and see if it’s perhaps crashing.
How are you deploying your Loki cluster? How many containers do you have for Loki? Can you share your configuration as well, please?
Hi Tony,
Thanks for the support!!!
Nothing on Loki is crashing but can see timeout on grafna pods…below is an example log
//logger=tsdb.loki endpoint=CheckHealth endpoint=queryData pluginId=loki dsName=loki dsUID=ae0ytjs1plclce uname=admin fromAlert=false t=2024-10-17T19:36:57.776731861Z level=info msg=“Response received from loki” duration=4.683155858s stage=databaseRequest statusCode=200 contentLength= start=2024-10-15T19:36:42.054Z end=2024-10-16T19:36:42.054Z step=24h0m0s query=“{pod="loki-test-gateway-6585df7f6c-46mct"} = **<mark>timeout</mark>**” queryType=range direction=backward maxLines=10 supportingQueryType=dataSample lokiHost=loki-test-gateway.loki-test.svc.cluster.local lokiPath=/loki/api/v1/query_range status=ok //
How are you deploying Loki cluster? We are deploying a simple scalable cluster using helm
How many containers do you have for loki ? 5 writer ; 5 readers ; 5 backend ; 2 grafana
my config below:
deploymentMode: SimpleScalable
loki:
readinessProbe:
httpGet:
path: /ready
port: http-metrics
initialDelaySeconds: 30
timeoutSeconds: 1
schemaConfig:
configs:
- from: 2024-04-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
image:
registry: docker.io
repository: grafana/loki
tag: main-0c39e15
revisionHistoryLimit: 10
enableServiceLinks: true
auth_enabled: false
storage:
type: s3
bucketNames:
chunks: loki-7boss-test
ruler: loki-7boss-test
admin: loki-7boss-test
s3:
s3: s3://loki-7boss-test
bucketNames: loki-7boss-test
endpoint: s3.us-east-1.amazonaws.com
region: us-east-1
secretAccessKey: xxxx
accessKeyId: xxxx
limits_config:
ingestion_rate_strategy: global
ingestion_rate_mb: 1000
ingestion_burst_size_mb: 1000
max_label_name_length: 10240
max_label_value_length: 20480
max_label_names_per_series: 300
reject_old_samples: true
reject_old_samples_max_age: 168h
creation_grace_period: 10m
max_streams_per_user: 0
max_line_size: 256000
max_entries_limit_per_query: 1000000
max_global_streams_per_user: 15000
max_chunks_per_query: 2000000
max_query_length: 721h
max_query_parallelism: 32
max_query_series: 500
cardinality_limit: 200000
max_streams_matchers_per_query: 1000
max_cache_freshness_per_query: 10m
per_stream_rate_limit: 512M
per_stream_rate_limit_burst: 1024M
split_queries_by_interval: 15m
tsdb_max_query_parallelism: 512
query_timeout: 150s
volume_enabled: true
volume_max_series: 1000
allow_structured_metadata: false
frontend:
scheduler_address: ‘{{ include “loki.querySchedulerAddress” . }}’
tail_proxy_url: ‘{{ include “loki.querierAddress” . }}’
frontend_worker:
scheduler_address: ‘{{ include “loki.querySchedulerAddress” . }}’
gateway:
enabled: true
service:
port: 80
type: ClusterIP
ingress:
enabled: true
ingressClassName: alb
annotations: {
“alb.ingress.kubernetes.io/certificate-arn”: “arn:aws:acm:us-east-1:296299459458:certificate/7abeea08-4ec1-467a-987e-c90b044d4ed6”,
“alb.ingress.kubernetes.io/healthcheck-path”: “/”,
“alb.ingress.kubernetes.io/listen-ports”: “[{"HTTPS": 443}]”,
“alb.ingress.kubernetes.io/scheme”: “internal”,
“alb.ingress.kubernetes.io/target-type”: “ip”,
“alb.ingress.kubernetes.io/subnets”: “subnet-02eff516b7c4be2c2,subnet-04e931bedf430a2de,subnet-08396f2327b4ba9e1”
}
hosts:
- host: loki-dev.ris-shared-dev.xyz.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: loki-gateway-tls
hosts:
- loki-dev.ris-shared-dev.xyz.com
backend:
replicas: 5
autoscaling:
enabled: false
terminationGracePeriodSeconds: 300
podManagementPolicy: “Parallel”
extraArgs:
- -config.expand-env=true
persistence:
volumeClaimsEnabled: false
storageClass: loki-ebs-sc
enableStatefulSetAutoDeletePVC: true
size: 10Gi
read:
replicas: 5
autoscaling:
enabled: false
targetModule: “read”
legacyReadTarget: false
terminationGracePeriodSeconds: 30
podManagementPolicy: “Parallel”
extraArgs:
- -config.expand-env=true
persistence:
enableStatefulSetAutoDeletePVC: false
write:
replicas: 5
autoscaling:
enabled: false
targetModule: “write”
podManagementPolicy: “Parallel”
extraArgs:
- -config.expand-env=true
persistence:
volumeClaimsEnabled: false
storageClass: loki-ebs-sc
enableStatefulSetAutoDeletePVC: true
size: 10Gi
singleBinary:
replicas: 0
ingester:
replicas: 0
querier:
replicas: 0
queryFrontend:
replicas: 0
queryScheduler:
replicas: 0
distributor:
replicas: 0
compactor:
replicas: 0
indexGateway:
replicas: 0
bloomCompactor:
replicas: 0
bloomGateway:
replicas: 0
we are using - simple scalable deployment via helm
containers - 5 reader ; 5 writer ; 5 backend & 2 grafana.
i dnt see loki crashing but the only thing i see is timeout on Grafana pod logs…
logger=tsdb.loki endpoint=CheckHealth endpoint=queryData pluginId=loki dsName=loki dsUID=ae0ytjs1plclce uname=admin fromAlert=false t=2024-10-17T19:36:57.776731861Z level=info msg=“Response received from loki” duration=4.683155858s stage=databaseRequest statusCode=200 contentLength= start=2024-10-15T19:36:42.054Z end=2024-10-16T19:36:42.054Z step=24h0m0s query=“{pod="loki-test-gateway-6585df7f6c-46mct"} |= timeout” queryType=range direction=backward maxLines=10 supportingQueryType=dataSample lokiHost=loki-test-gateway.loki-test.svc.cluster.local lokiPath=/loki/api/v1/query_range status=ok
Failed to load log volume for this query
502 Bad Gateway
502 Bad Gateway
nginx/1.27.2
just got this error…
Can anyone help on how to improve histogram performance? I’m not able to find documentation around histogram as well. thanks in advance
