Tempo with TLS configuration return "Connection reset by peer in connection"

timec80 · June 2, 2023, 12:02pm

I’ve configured Tempo instance (running latest version via docker distribution) with TLS, using instructions at Configure TLS communication | Grafana Tempo documentation

The certificate is valid wildcard certificate
grpc (port 9095, with TLS) connector : is working
receivers.otlp (http with TLS): is working
http (port 3200, with TLS) connector : is not working. Executing “curl” to this endpoint give the following output:

*   Trying <redacted>:3200...
* Connected to <redacted> (<redacted>) port 3200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to <redacted>:3200
* Closing connection 0
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to <redacted>:3200

Here is the config file:

server:

  http_listen_port: 3200
  grpc_listen_port: 9095

  http_tls_config:
    cert_file: /etc/tempo/tls.crt
    key_file: /etc/tempo/tls.key
    client_auth_type: NoClientCert
  grpc_tls_config:
    cert_file: /etc/tempo/tls.crt
    key_file: /etc/tempo/tls.key
    client_auth_type: NoClientCert

ingester_client:
  grpc_client_config:
    tls_enabled: true
    tls_cert_path: /etc/tempo/tls.crt
    tls_key_path: /etc/tempo/tls.key
    tls_insecure_skip_verify: true

metrics_generator_client:
  grpc_client_config:
    tls_enabled: true
    tls_cert_path: /etc/tempo/tls.crt
    tls_key_path: /etc/tempo/tls.key
    tls_insecure_skip_verify: true

querier:
  frontend_worker:
    frontend_address: 127.0.0.1:9095
    grpc_client_config:
      tls_enabled: true
      tls_cert_path: /etc/tempo/tls.crt
      tls_key_path: /etc/tempo/tls.key
      tls_insecure_skip_verify: true


memberlist:
  tls_enabled: true
  tls_cert_path: /etc/tempo/tls.crt
  tls_key_path: /etc/tempo/tls.key
  tls_insecure_skip_verify: true

distributor:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: 0.0.0.0:3201
          tls:
            cert_file: /etc/tempo/tls.crt
            key_file: /etc/tempo/tls.key

        grpc:
          endpoint: 0.0.0.0:3202
          tls:
            cert_file: /etc/tempo/tls.crt
            key_file: /etc/tempo/tls.key
compactor:
  compaction:
    block_retention: 6h

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: some-cluster
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
      - url: https://<redacted>:9090/api/v1/write
        send_exemplars: true

storage:
  trace:
    backend: local
    wal:
      path: /tmp/tempo/wal
    local:
      path: /tmp/tempo/blocks
overrides:
  metrics_generator_processors: [service-graphs, span-metrics]

Has somebody any thoughts ?

Thank you

surajsidh · June 5, 2023, 11:06am

can you try removing http_tls_config and see if you reach it without TLS config?

also, can you share tempo debug logs from the time range when you see this connection reset??

also see: Xml and More: End-to-End Security―How to Debug SSL/TLS Issues for more on debugging TLS issues.

timec80 · June 5, 2023, 1:32pm

Hello, thank you for your reply.

Without http_tls_config everything is working as expected (pushing spans and Grafana connection).

With http_tls_config enabled, neither pushing spans nor Grafana connection is working.
And curl operation gives “Connection reset by peer”.

Here, the Tempo’s output with log_level: debug. This log covers the startup sequence and time range of curl’s operation.

level=info ts=2023-06-05T13:22:12.466282983Z caller=main.go:215 msg="initialising OpenTracing tracer"
level=info ts=2023-06-05T13:22:12.468171296Z caller=main.go:102 msg="Starting Tempo" version="(version=main-890838b, branch=main, revision=890838bf6)"
level=info ts=2023-06-05T13:22:12.471969088Z caller=server.go:354 http=[::]:3200 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2023-06-05T13:22:12.474147868Z caller=worker.go:103 msg="Starting querier worker connected to query-frontend" frontend=127.0.0.1:9095
ts=2023-06-05T13:22:12Z level=info msg="OTel Shim Logger Initialized" component=tempo
level=info ts=2023-06-05T13:22:12.481856482Z caller=frontend.go:45 msg="creating middleware in query frontend"
level=info ts=2023-06-05T13:22:12.482464707Z caller=tempodb.go:429 msg="polling enabled" interval=5m0s concurrency=50
level=info ts=2023-06-05T13:22:12.483194623Z caller=poller.go:175 msg="successfully pulled tenant index" tenant=single-tenant createdAt=2023-06-05T13:21:35.310093904Z metas=1 compactedMetas=2
level=info ts=2023-06-05T13:22:12.483249365Z caller=poller.go:128 msg="blocklist poll complete" seconds=0.000457821
level=debug ts=2023-06-05T13:22:12.484030452Z caller=module_service.go:72 msg="module waiting for initialization" module=metrics-generator waiting_for=internal-server
level=debug ts=2023-06-05T13:22:12.484212963Z caller=module_service.go:72 msg="module waiting for initialization" module=overrides waiting_for=internal-server
level=debug ts=2023-06-05T13:22:12.484250591Z caller=module_service.go:72 msg="module waiting for initialization" module=memberlist-kv waiting_for=internal-server
level=debug ts=2023-06-05T13:22:12.484399608Z caller=module_service.go:72 msg="module waiting for initialization" module=usage-report waiting_for=internal-server
level=debug ts=2023-06-05T13:22:12.484427351Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=internal-server
level=debug ts=2023-06-05T13:22:12.484563671Z caller=module_service.go:72 msg="module waiting for initialization" module=query-frontend waiting_for=overrides
level=info ts=2023-06-05T13:22:12.484586578Z caller=module_service.go:82 msg=initialising module=internal-server
level=info ts=2023-06-05T13:22:12.484711553Z caller=module_service.go:82 msg=initialising module=store
level=debug ts=2023-06-05T13:22:12.484752358Z caller=module_service.go:72 msg="module waiting for initialization" module=ring waiting_for=internal-server
level=debug ts=2023-06-05T13:22:12.484771806Z caller=module_service.go:72 msg="module waiting for initialization" module=ring waiting_for=memberlist-kv
level=debug ts=2023-06-05T13:22:12.484785182Z caller=module_service.go:72 msg="module waiting for initialization" module=server waiting_for=internal-server
level=info ts=2023-06-05T13:22:12.484917846Z caller=module_service.go:82 msg=initialising module=server
level=debug ts=2023-06-05T13:22:12.48510224Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=server
level=debug ts=2023-06-05T13:22:12.485135602Z caller=module_service.go:72 msg="module waiting for initialization" module=compactor waiting_for=server
level=debug ts=2023-06-05T13:22:12.48522291Z caller=module_service.go:72 msg="module waiting for initialization" module=metrics-generator-ring waiting_for=internal-server
level=debug ts=2023-06-05T13:22:12.485271848Z caller=module_service.go:72 msg="module waiting for initialization" module=metrics-generator-ring waiting_for=memberlist-kv
level=debug ts=2023-06-05T13:22:12.485297458Z caller=module_service.go:72 msg="module waiting for initialization" module=querier waiting_for=store
level=debug ts=2023-06-05T13:22:12.485423274Z caller=module_service.go:72 msg="module waiting for initialization" module=querier waiting_for=usage-report
level=debug ts=2023-06-05T13:22:12.485451174Z caller=module_service.go:72 msg="module waiting for initialization" module=metrics-generator waiting_for=memberlist-kv
level=debug ts=2023-06-05T13:22:12.485569998Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=memberlist-kv
level=debug ts=2023-06-05T13:22:12.485586268Z caller=module_service.go:72 msg="module waiting for initialization" module=usage-report waiting_for=memberlist-kv
level=debug ts=2023-06-05T13:22:12.485598083Z caller=module_service.go:72 msg="module waiting for initialization" module=memberlist-kv waiting_for=server
level=debug ts=2023-06-05T13:22:12.485608782Z caller=module_service.go:72 msg="module waiting for initialization" module=overrides waiting_for=server
level=info ts=2023-06-05T13:22:12.486615048Z caller=module_service.go:82 msg=initialising module=overrides
level=info ts=2023-06-05T13:22:12.486672131Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=debug ts=2023-06-05T13:22:12.486724601Z caller=module_service.go:72 msg="module waiting for initialization" module=compactor waiting_for=store
level=debug ts=2023-06-05T13:22:12.486739155Z caller=module_service.go:72 msg="module waiting for initialization" module=compactor waiting_for=usage-report
level=debug ts=2023-06-05T13:22:12.486750184Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=store
level=debug ts=2023-06-05T13:22:12.486758745Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=usage-report
level=debug ts=2023-06-05T13:22:12.486768641Z caller=module_service.go:72 msg="module waiting for initialization" module=query-frontend waiting_for=server
level=debug ts=2023-06-05T13:22:12.486777331Z caller=module_service.go:72 msg="module waiting for initialization" module=query-frontend waiting_for=store
level=debug ts=2023-06-05T13:22:12.486785994Z caller=module_service.go:72 msg="module waiting for initialization" module=query-frontend waiting_for=usage-report
level=debug ts=2023-06-05T13:22:12.486795798Z caller=module_service.go:72 msg="module waiting for initialization" module=usage-report waiting_for=server
level=info ts=2023-06-05T13:22:12.486805405Z caller=module_service.go:82 msg=initialising module=usage-report
level=debug ts=2023-06-05T13:22:12.487006967Z caller=mock.go:150 msg=Get key=collectors/usagestats_token wait_index=0
level=debug ts=2023-06-05T13:22:12.487045009Z caller=mock.go:157 msg="Get - not found" key=collectors/usagestats_token
level=debug ts=2023-06-05T13:22:12.487468386Z caller=mock.go:123 msg=CAS key=collectors/usagestats_token modify_index=0 value="\"{\\\"UID\\\":\\\"15b8120e-f347-4264-bc6b-cf8d3912\""
level=debug ts=2023-06-05T13:22:12.487544422Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=metrics-generator-ring
level=debug ts=2023-06-05T13:22:12.487564708Z caller=module_service.go:72 msg="module waiting for initialization" module=metrics-generator waiting_for=overrides
level=debug ts=2023-06-05T13:22:12.48757678Z caller=module_service.go:72 msg="module waiting for initialization" module=metrics-generator waiting_for=server
level=debug ts=2023-06-05T13:22:12.487600275Z caller=module_service.go:72 msg="module waiting for initialization" module=metrics-generator waiting_for=usage-report
level=info ts=2023-06-05T13:22:12.487611011Z caller=module_service.go:82 msg=initialising module=metrics-generator
level=debug ts=2023-06-05T13:22:12.488028772Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=internal-server
level=debug ts=2023-06-05T13:22:12.488050643Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=memberlist-kv
level=debug ts=2023-06-05T13:22:12.488060597Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=overrides
level=info ts=2023-06-05T13:22:12.488070095Z caller=module_service.go:82 msg=initialising module=ingester
level=info ts=2023-06-05T13:22:12.488084277Z caller=ingester.go:327 msg="beginning wal replay"
level=warn ts=2023-06-05T13:22:12.488189987Z caller=wal.go:112 msg="unowned file entry ignored during wal replay" file=blocks err=null
level=info ts=2023-06-05T13:22:12.488216246Z caller=ingester.go:365 msg="wal replay complete"
level=info ts=2023-06-05T13:22:12.488390421Z caller=ingester.go:379 msg="reloading local blocks" tenants=0
level=info ts=2023-06-05T13:22:12.48855195Z caller=lifecycler.go:570 msg="not loading tokens from file, tokens file path is empty"
level=debug ts=2023-06-05T13:22:12.488619888Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=0
level=debug ts=2023-06-05T13:22:12.488656838Z caller=mock.go:157 msg="Get - not found" key=collectors/ring
level=info ts=2023-06-05T13:22:12.488766516Z caller=lifecycler.go:599 msg="instance not found in ring, adding with no tokens" ring=ingester
level=debug ts=2023-06-05T13:22:12.488832533Z caller=mock.go:123 msg=CAS key=collectors/ring modify_index=0 value="\"0\\xbc\\n.\\n\\f37f71c86ca6d\\x12\\x1e\\n\\x0e127.0.0.1:9095\\x10\\x84\\xc3\\xf7\""
level=debug ts=2023-06-05T13:22:12.489088647Z caller=lifecycler.go:435 msg="JoinAfter expired" ring=ingester
level=info ts=2023-06-05T13:22:12.489169551Z caller=lifecycler.go:439 msg="auto-joining cluster after timeout" ring=ingester
level=debug ts=2023-06-05T13:22:12.489226691Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=0
level=debug ts=2023-06-05T13:22:12.489250129Z caller=mock.go:196 msg=Get key=collectors/ring modify_index=3 value="\"0\\xbc\\n.\\n\\f37f71c86ca6d\\x12\\x1e\\n\\x0e127.0.0.1:9095\\x10\\x84\\xc3\\xf7\""
level=debug ts=2023-06-05T13:22:12.489604631Z caller=mock.go:123 msg=CAS key=collectors/ring modify_index=3 value="\"\\xb1\\x05\\xf4\\xb0\\x02\\n\\xae\\x05\\n\\f37f71c86ca6d\\x12\\x9d\\x05\\n\\x0e127.0.0.1:909\""
level=debug ts=2023-06-05T13:22:12.489663578Z caller=module_service.go:72 msg="module waiting for initialization" module=metrics-generator-ring waiting_for=server
level=info ts=2023-06-05T13:22:12.489678221Z caller=module_service.go:82 msg=initialising module=metrics-generator-ring
level=debug ts=2023-06-05T13:22:12.489715253Z caller=mock.go:150 msg=Get key=collectors/metrics-generator wait_index=0
level=debug ts=2023-06-05T13:22:12.489728406Z caller=mock.go:157 msg="Get - not found" key=collectors/metrics-generator
level=info ts=2023-06-05T13:22:12.489748864Z caller=ring.go:262 msg="ring doesn't exist in KV store yet"
level=debug ts=2023-06-05T13:22:12.489799747Z caller=mock.go:150 msg=Get key=collectors/metrics-generator wait_index=0
level=debug ts=2023-06-05T13:22:12.489840998Z caller=mock.go:157 msg="Get - not found" key=collectors/metrics-generator
level=debug ts=2023-06-05T13:22:12.489855114Z caller=client.go:255 msg="value is nil" key=collectors/metrics-generator index=4
level=debug ts=2023-06-05T13:22:12.489982265Z caller=mock.go:150 msg=Get key=collectors/metrics-generator wait_index=4
level=debug ts=2023-06-05T13:22:12.490007602Z caller=module_service.go:72 msg="module waiting for initialization" module=ring waiting_for=server
level=info ts=2023-06-05T13:22:12.490020198Z caller=module_service.go:82 msg=initialising module=ring
level=debug ts=2023-06-05T13:22:12.490055349Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=0
level=debug ts=2023-06-05T13:22:12.490072587Z caller=mock.go:196 msg=Get key=collectors/ring modify_index=4 value="\"\\xb1\\x05\\xf4\\xb0\\x02\\n\\xae\\x05\\n\\f37f71c86ca6d\\x12\\x9d\\x05\\n\\x0e127.0.0.1:909\""
level=debug ts=2023-06-05T13:22:12.490395861Z caller=mock.go:150 msg=Get key=collectors/metrics-generator wait_index=0
level=debug ts=2023-06-05T13:22:12.490539658Z caller=module_service.go:72 msg="module waiting for initialization" module=compactor waiting_for=internal-server
level=debug ts=2023-06-05T13:22:12.490655778Z caller=module_service.go:72 msg="module waiting for initialization" module=compactor waiting_for=memberlist-kv
level=debug ts=2023-06-05T13:22:12.490673902Z caller=module_service.go:72 msg="module waiting for initialization" module=compactor waiting_for=overrides
level=info ts=2023-06-05T13:22:12.490685147Z caller=module_service.go:82 msg=initialising module=compactor
level=info ts=2023-06-05T13:22:12.490717594Z caller=tempodb.go:429 msg="polling enabled" interval=5m0s concurrency=50
level=debug ts=2023-06-05T13:22:12.490447638Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=0
level=debug ts=2023-06-05T13:22:12.490477015Z caller=module_service.go:72 msg="module waiting for initialization" module=query-frontend waiting_for=internal-server
level=debug ts=2023-06-05T13:22:12.491394064Z caller=module_service.go:72 msg="module waiting for initialization" module=query-frontend waiting_for=memberlist-kv
level=info ts=2023-06-05T13:22:12.491582413Z caller=module_service.go:82 msg=initialising module=query-frontend
level=debug ts=2023-06-05T13:22:12.490509765Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=overrides
level=debug ts=2023-06-05T13:22:12.492124324Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=ring
level=debug ts=2023-06-05T13:22:12.490530372Z caller=module_service.go:72 msg="module waiting for initialization" module=querier waiting_for=internal-server
level=debug ts=2023-06-05T13:22:12.490603578Z caller=mock.go:157 msg="Get - not found" key=collectors/metrics-generator
level=info ts=2023-06-05T13:22:12.49197686Z caller=poller.go:199 msg="writing tenant index" tenant=single-tenant metas=1 compactedMetas=2
level=debug ts=2023-06-05T13:22:12.492505745Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=server
level=debug ts=2023-06-05T13:22:12.493036176Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=usage-report
level=info ts=2023-06-05T13:22:12.493059099Z caller=module_service.go:82 msg=initialising module=distributor
level=debug ts=2023-06-05T13:22:12.492654557Z caller=module_service.go:72 msg="module waiting for initialization" module=querier waiting_for=memberlist-kv
level=debug ts=2023-06-05T13:22:12.493280826Z caller=module_service.go:72 msg="module waiting for initialization" module=querier waiting_for=metrics-generator-ring
level=debug ts=2023-06-05T13:22:12.49331798Z caller=module_service.go:72 msg="module waiting for initialization" module=querier waiting_for=overrides
level=debug ts=2023-06-05T13:22:12.493406129Z caller=module_service.go:72 msg="module waiting for initialization" module=querier waiting_for=ring
level=debug ts=2023-06-05T13:22:12.493497011Z caller=module_service.go:72 msg="module waiting for initialization" module=querier waiting_for=server
level=info ts=2023-06-05T13:22:12.493537722Z caller=module_service.go:82 msg=initialising module=querier
level=info ts=2023-06-05T13:22:12.492828293Z caller=basic_lifecycler.go:261 msg="instance not found in the ring" instance=37f71c86ca6d ring=metrics-generator
level=debug ts=2023-06-05T13:22:12.49437337Z caller=mock.go:123 msg=CAS key=collectors/metrics-generator modify_index=0 value="\"\\xa6\\n\\xf4%\\x05\\n\\xa3\\n\\n\\f37f71c86ca6d\\x12\\x92\\n\\n\\x0e127.0.0.1:909\""
level=debug ts=2023-06-05T13:22:12.492855343Z caller=mock.go:196 msg=Get key=collectors/ring modify_index=4 value="\"\\xb1\\x05\\xf4\\xb0\\x02\\n\\xae\\x05\\n\\f37f71c86ca6d\\x12\\x9d\\x05\\n\\x0e127.0.0.1:909\""
level=debug ts=2023-06-05T13:22:12.494460131Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=4
level=info ts=2023-06-05T13:22:12.494564063Z caller=worker.go:179 msg="adding connection" addr=127.0.0.1:9095
level=debug ts=2023-06-05T13:22:12.495597539Z caller=mock.go:196 msg=Get key=collectors/metrics-generator modify_index=5 value="\"\\xa6\\n\\xf4%\\x05\\n\\xa3\\n\\n\\f37f71c86ca6d\\x12\\x92\\n\\n\\x0e127.0.0.1:909\""
level=debug ts=2023-06-05T13:22:12.495807086Z caller=mock.go:150 msg=Get key=collectors/metrics-generator wait_index=5
ts=2023-06-05T13:22:12Z level=warn msg="Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks" component=tempo documentation=https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks
ts=2023-06-05T13:22:12Z level=info msg="Starting GRPC server" component=tempo endpoint=0.0.0.0:3202
ts=2023-06-05T13:22:12Z level=warn msg="Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks" component=tempo documentation=https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks
ts=2023-06-05T13:22:12Z level=info msg="Starting HTTP server" component=tempo endpoint=0.0.0.0:3201
level=info ts=2023-06-05T13:22:12.498776407Z caller=poller.go:128 msg="blocklist poll complete" seconds=0.008037108
level=info ts=2023-06-05T13:22:12.499054458Z caller=compactor.go:162 msg="enabling compaction"
level=info ts=2023-06-05T13:22:12.499298062Z caller=tempodb.go:401 msg="compaction and retention enabled."
level=info ts=2023-06-05T13:22:12.499641632Z caller=app.go:194 msg="Tempo started"
level=debug ts=2023-06-05T13:22:14.474458394Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/metrics-generator
level=debug ts=2023-06-05T13:22:14.474542381Z caller=mock.go:150 msg=Get key=collectors/metrics-generator wait_index=5
level=debug ts=2023-06-05T13:22:14.474568397Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
level=debug ts=2023-06-05T13:22:14.474586077Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=4
level=debug ts=2023-06-05T13:22:15.474769246Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
level=debug ts=2023-06-05T13:22:15.474829768Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=4
level=debug ts=2023-06-05T13:22:15.474847179Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/metrics-generator
level=debug ts=2023-06-05T13:22:15.474860168Z caller=mock.go:150 msg=Get key=collectors/metrics-generator wait_index=5
level=debug ts=2023-06-05T13:22:16.475254837Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/metrics-generator
level=debug ts=2023-06-05T13:22:16.475366379Z caller=mock.go:150 msg=Get key=collectors/metrics-generator wait_index=5
level=debug ts=2023-06-05T13:22:16.475391282Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
level=debug ts=2023-06-05T13:22:16.475621035Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=4
level=debug ts=2023-06-05T13:22:17.47612394Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
level=debug ts=2023-06-05T13:22:17.476262175Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=4
level=debug ts=2023-06-05T13:22:17.476286504Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/metrics-generator
level=debug ts=2023-06-05T13:22:17.476328013Z caller=mock.go:150 msg=Get key=collectors/metrics-generator wait_index=5
level=debug ts=2023-06-05T13:22:17.48838158Z caller=mock.go:150 msg=Get key=collectors/usagestats_token wait_index=0
level=debug ts=2023-06-05T13:22:17.48850077Z caller=mock.go:196 msg=Get key=collectors/usagestats_token modify_index=2 value="\"{\\\"UID\\\":\\\"15b8120e-f347-4264-bc6b-cf8d3912\""
level=debug ts=2023-06-05T13:22:17.489892049Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=0
level=debug ts=2023-06-05T13:22:17.489946313Z caller=mock.go:196 msg=Get key=collectors/ring modify_index=4 value="\"\\xb1\\x05\\xf4\\xb0\\x02\\n\\xae\\x05\\n\\f37f71c86ca6d\\x12\\x9d\\x05\\n\\x0e127.0.0.1:909\""
level=debug ts=2023-06-05T13:22:17.49001718Z caller=mock.go:123 msg=CAS key=collectors/ring modify_index=4 value="\"\\xb1\\x05\\xf4\\xb0\\x02\\n\\xae\\x05\\n\\f37f71c86ca6d\\x12\\x9d\\x05\\n\\x0e127.0.0.1:909\""
level=debug ts=2023-06-05T13:22:17.490047002Z caller=mock.go:196 msg=Get key=collectors/ring modify_index=6 value="\"\\xb1\\x05\\xf4\\xb0\\x02\\n\\xae\\x05\\n\\f37f71c86ca6d\\x12\\x9d\\x05\\n\\x0e127.0.0.1:909\""
level=debug ts=2023-06-05T13:22:17.49007318Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=6
level=debug ts=2023-06-05T13:22:17.49557676Z caller=mock.go:150 msg=Get key=collectors/metrics-generator wait_index=0
level=debug ts=2023-06-05T13:22:17.495664908Z caller=mock.go:196 msg=Get key=collectors/metrics-generator modify_index=5 value="\"\\xa6\\n\\xf4%\\x05\\n\\xa3\\n\\n\\f37f71c86ca6d\\x12\\x92\\n\\n\\x0e127.0.0.1:909\""
level=debug ts=2023-06-05T13:22:17.495754126Z caller=mock.go:123 msg=CAS key=collectors/metrics-generator modify_index=5 value="\"\\xa6\\n\\xf4%\\x05\\n\\xa3\\n\\n\\f37f71c86ca6d\\x12\\x92\\n\\n\\x0e127.0.0.1:909\""
level=debug ts=2023-06-05T13:22:17.495788567Z caller=mock.go:196 msg=Get key=collectors/metrics-generator modify_index=7 value="\"\\xa6\\n\\xf4%\\x05\\n\\xa3\\n\\n\\f37f71c86ca6d\\x12\\x92\\n\\n\\x0e127.0.0.1:909\""
level=debug ts=2023-06-05T13:22:17.495822249Z caller=mock.go:150 msg=Get key=collectors/metrics-generator wait_index=7

Thank you

surajsidh · June 6, 2023, 4:25pm

logs look normal to me, I would recommend looking into networking and making sure that requests are reaching tempo and everything has correct config and certs