Liveness probe failed: HTTP probe failed with statuscode: 400

  • What Grafana version and what operating system are you using?
    11.4.0

  • What are you trying to achieve?
    Grafana working on HTTPS. Its behind nginx ingress controller.

  • How are you trying to achieve it?
    Kubernetes with Helm. Mounted configmap as volume with ssl key and certificate. Tried with crt and pem. Edited the deployment for mounting certificate

  • What happened?
    when described the pod
    Liveness probe failed: HTTP probe failed with statuscode: 400
    Readiness probe failed: HTTP probe failed with statuscode: 400

Logs from the pod
“logger=featuremgmt t=2024-12-27T10:22:42.120869958Z level=info msg=FeatureToggles panelMonitoring=true pinNavItems=true logRowsPopoverMenu=true prometheusAzureOverrideAudience=true prometheusConfigOverhaulAuth=true recordedQueriesMulti=true cloudWatchNewLabelParsing=true alertingNoDataErrorExecution=true topnav=true logsInfiniteScrolling=true awsAsyncQueryCaching=true lokiQuerySplitting=true publicDashboards=true accessControlOnCall=true lokiQueryHints=true tlsMemcached=true angularDeprecationUI=true dashboardSceneForViewers=true cloudWatchCrossAccountQuerying=true transformationsVariableSupport=true influxdbBackendMigration=true dataplaneFrontendFallback=true ssoSettingsApi=true alertingSimplifiedRouting=true dashgpt=true publicDashboardsScene=true nestedFolders=true annotationPermissionUpdate=true formatString=true addFieldFromCalculationStatFunctions=true managedPluginsInstall=true prometheusMetricEncyclopedia=true alertingInsights=true autoMigrateXYChartPanel=true lokiMetricDataplane=true logsContextDatasourceUi=true correlations=true recoveryThreshold=true kubernetesPlaylists=true promQLScope=true groupToNestedTableTransformation=true logsExploreTableVisualisation=true openSearchBackendFlowEnabled=true notificationBanner=true dashboardScene=true transformationsRedesign=true exploreMetrics=true lokiStructuredMetadata=true cloudWatchRoundUpEndTime=true dashboardSceneSolo=true
logger=sqlstore t=2024-12-27T10:22:42.120945969Z level=info msg=“Connecting to DB” dbtype=sqlite3
logger=sqlstore t=2024-12-27T10:22:42.120965302Z level=warn msg=“SQLite database file has broader permissions than it should” path=/var/lib/grafana/grafana.db mode=-rw-rw---- expected=-rw-r-----
logger=migrator t=2024-12-27T10:22:42.141247297Z level=info msg=“Locking database”
logger=migrator t=2024-12-27T10:22:42.141288979Z level=info msg=“Starting DB migrations”
logger=migrator t=2024-12-27T10:22:42.171848708Z level=info msg=“migrations completed” performed=0 skipped=611 duration=1.684055ms
logger=migrator t=2024-12-27T10:22:42.172675641Z level=info msg=“Unlocking database”
logger=secrets t=2024-12-27T10:22:42.173270728Z level=info msg=“Envelope encryption state” enabled=true currentprovider=secretKey.v1
logger=plugin.angulardetectorsprovider.dynamic t=2024-12-27T10:22:42.28336603Z level=info msg=“Restored cache from database” duration=569.873µs
logger=plugin.store t=2024-12-27T10:22:42.286261535Z level=info msg=“Loading plugins…”
logger=plugins.registration t=2024-12-27T10:22:42.338684344Z level=error msg=“Could not register plugin” pluginId=xychart error=“plugin xychart is already registered”
logger=plugins.initialization t=2024-12-27T10:22:42.338775445Z level=error msg=“Could not initialize plugin” pluginId=xychart error=“plugin xychart is already registered”
logger=local.finder t=2024-12-27T10:22:42.339163506Z level=warn msg=“Skipping finding plugins as directory does not exist” path=/usr/share/grafana/plugins-bundled
logger=plugins.registration t=2024-12-27T10:22:42.441751107Z level=info msg=“Plugin registered” pluginId=grafana-lokiexplore-app
logger=plugin.store t=2024-12-27T10:22:42.441787087Z level=info msg=“Plugins loaded” count=55 duration=155.526397ms
logger=query_data t=2024-12-27T10:22:42.462827226Z level=info msg=“Query Service initialization”
logger=live.push_http t=2024-12-27T10:22:42.475441098Z level=info msg=“Live Push Gateway initialization”
logger=ngalert.notifier.alertmanager org=1 t=2024-12-27T10:22:42.484622254Z level=info msg=“Applying new configuration to Alertmanager” configHash=d2c56faca6af2a5772ff4253222f7386
logger=ngalert.state.manager t=2024-12-27T10:22:42.50058071Z level=info msg=“Running in alternative execution of Error/NoData mode”
logger=infra.usagestats.collector t=2024-12-27T10:22:42.503202593Z level=info msg=“registering usage stat providers” usageStatsProvidersLen=2
logger=provisioning.alerting t=2024-12-27T10:22:42.504004209Z level=info msg=“starting to provision alerting”
logger=provisioning.alerting t=2024-12-27T10:22:42.504027377Z level=info msg=“finished to provision alerting”
logger=http.server t=2024-12-27T10:22:42.506026119Z level=info msg=“HTTP Server TLS settings” scheme=https MinTLSVersion=TLS1.2 configuredciphers=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA
logger=http.server t=2024-12-27T10:22:42.506353303Z level=info msg=“HTTP Server Listen” address=[::]:3000 protocol=https subUrl= socket=
logger=grafanaStorageLogger t=2024-12-27T10:22:42.506674794Z level=info msg=“Storage starting”
logger=ngalert.state.manager t=2024-12-27T10:22:42.506660052Z level=info msg=“Warming state cache for startup”
logger=ngalert.state.manager t=2024-12-27T10:22:42.507087151Z level=info msg=“State cache has been initialized” states=0 duration=426.784µs
logger=ngalert.multiorg.alertmanager t=2024-12-27T10:22:42.527411632Z level=info msg=“Starting MultiOrg Alertmanager”
logger=provisioning.dashboard t=2024-12-27T10:22:42.52794558Z level=info msg=“starting to provision dashboards”
logger=provisioning.dashboard t=2024-12-27T10:22:42.527989952Z level=info msg=“finished to provision dashboards”
logger=ngalert.scheduler t=2024-12-27T10:22:42.528312721Z level=info msg=“Starting scheduler” tickInterval=10s maxAttempts=1
logger=ticker t=2024-12-27T10:22:42.528386008Z level=info msg=starting first_tick=2024-12-27T10:22:50Z
logger=grafana.update.checker t=2024-12-27T10:22:42.598578821Z level=info msg=“Update check succeeded” duration=73.092163ms
logger=plugins.update.checker t=2024-12-27T10:22:42.603635861Z level=info msg=“Update check succeeded” duration=76.872206ms
logger=grafana-apiserver t=2024-12-27T10:22:43.078656657Z level=info msg=“Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager”
logger=grafana-apiserver t=2024-12-27T10:22:43.079270978Z level=info msg=“Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager”
logger=grafana-apiserver t=2024-12-27T10:22:43.080826192Z level=info msg=“Adding GroupVersion iam.grafana.app v0alpha1 to ResourceManager”
t=2024-12-27T10:22:43.439333931Z level=info msg=“http: TLS handshake error from 172.31.95.231:56492: client sent an HTTP request to an HTTPS server”
t=2024-12-27T10:23:31.769822772Z level=info msg=“http: TLS handshake error from 172.31.95.231:58444: client sent an HTTP request to an HTTPS server”
t=2024-12-27T10:24:10.843917614Z level=info msg=“http: TLS handshake error from 172.31.95.231:34798: client sent an HTTP request to an HTTPS server”
logger=infra.usagestats t=2024-12-27T10:24:16.533215604Z level=info msg=“Usage stats are ready to report”
t=2024-12-27T10:25:11.770129726Z level=info msg=“http: TLS handshake error from 172.31.95.231:49108: client sent an HTTP request to an HTTPS server”
t=2024-12-27T10:25:22.842914618Z level=info msg=“http: TLS handshake error from 172.31.95.231:59234: client sent an HTTP request to an HTTPS server””

  • What did you expect to happen?
    grafana should be running on https

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    shared above

  • Did you follow any online instructions? If so, what is the URL?
    Set up Grafana HTTPS for secure web traffic | Grafana documentation the ssl is real time

Your probes are making HTTP request, but your Grafana setup accepts only HTTPS requests (so it doesn’t understand HTTP requests).

Configure your probes to make https requests.

Generally, non paranoid K8s deployments are not using TLS on the Grafana pod side - they use TLS on the ingress/service level, which is doing TLS offloading and then Grafana pod will receive plain HTTP requests. Ask your K8S admin about these options.

Thanks jangaraj.
Can you help me to do these settings ? as we are new to k8s and need to setup the grafana with https. It would be great if you guide. :slight_smile: