Loki S3 Storage Handshake TLS Failure

Grafana Loki
1

Hi all, we on going explore use Loki with Object Storage S3 Compatible, we use HCP Object Storage and found some error like this :

level=info ts=2025-02-18T10:59:21.9780826Z caller=compactor.go:527 msg="compactor startup delay completed"
level=error ts=2025-02-18T10:59:21.983317Z caller=cached_client.go:189 msg="failed to build table names cache" err="RequestError: send request failed\ncaused by: Get \"https://tn-eureka.devhcp01.dti.co.id/ns-corebackendmonitoring%2Focp-chunks?delimiter=%2F&list-type=2&prefix=index%2F\": remote error: tls: handshake failure"
level=error ts=2025-02-18T10:59:21.9878689Z caller=cached_client.go:189 msg="failed to build table names cache" err="RequestError: send request failed\ncaused by: Get \"https://tn-eureka.devhcp01.dti.co.id/ns-corebackendmonitoring%2Focp-chunks?delimiter=%2F&list-type=2&prefix=index%2F\": remote error: tls: handshake failure"
level=error ts=2025-02-18T10:59:21.9903072Z caller=cached_client.go:189 msg="failed to build table names cache" err="RequestError: send request failed\ncaused by: Get \"https://tn-eureka.devhcp01.dti.co.id/ns-corebackendmonitoring%2Focp-chunks?delimiter=%2F&list-type=2&prefix=index%2F\": remote error: tls: handshake failure"
level=error ts=2025-02-18T10:59:21.9903072Z caller=compactor.go:534 msg="failed to run compaction" err="failed to list tables: RequestError: send request failed\ncaused by: Get \"https://tn-eureka.devhcp01.dti.co.id/ns-corebackendmonitoring%2Focp-chunks?delimiter=%2F&list-type=2&prefix=index%2F\": remote error: tls: handshake failure"
level=info ts=2025-02-18T10:59:21.9903072Z caller=compactor.go:592 msg="compactor started"

and here config.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: debug
  grpc_server_max_concurrent_streams: 1000
  
common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  replication_factor: 1
  storage:
    s3:
      bucketnames: ns-corebackendmonitoring/ocp-chunks
      region: grha
      endpoint: https://tn-eureka.devhcp01.dti.co.id/
      access_key_id: bW9uaXRvcmwluZ2NvcmViYWNrZW5wkX2NvbnRyaWJw1dG9y
      s3forcepathstyle: false
      secret_access_key: 87223837b1443b111d2028d8315e41668bb6
      insecure: false
      http_config:
        ca_file: ca-file.pem
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

limits_config:
  metric_aggregation_enabled: true
  
storage_config:
  tsdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h
  
pattern_ingester:
  enabled: true
  metric_aggregation:
    loki_address: localhost:3100

ruler:
  alertmanager_url: http://localhost:9093

frontend:
  encoding: protobuf

is there any clue how to check tls in loki s3 object storage ?

2 
openssl s_client -connect tn-eureka.devhcp01.dti.co.id:443
3

Unrelated, but you have your secrets in the config section.

1 Like
4

thank you for your solution, it’s kind get information from S3 Server, and we want to know detail communication loki with s3 server until show error that, do you know how to do it?

5

thank you for your suggestion, next config we put into secret and not in Helm

6

That’s not a solution, but a clue how to check tls and you didn’t show result.

7

here is some output for that

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: DH, 2048 bits
---
SSL handshake has read 3907 bytes and written 697 bytes
Verification error: unable to verify the first certificate
---
New, SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 7D00D50C7B7014DCDD1E0F7FC850407B2835027C6A9D19B57916F646F489B508
    Session-ID-ctx:
    Master-Key: 1494FCFF4347D961D2C813954EA5AB96D804C3FA57FC942C783BFD2FC990A3D42F4C9522B75B1328548DEC65FF83BAE9
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1740460411
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: yes

I think because we don’t have CA for s3 server in container loki, how to inject custom CA to loki container ?

i already try use http_config ( ca_file ) but it’s not work

8

If you need custom CA I am pretty sure you need to build it in. Use Loki’s container as base and add your CA cert (and run whatever command for your OS to update/refresh CA cert).