Can't configure dashboards in grafana running in kubernetes

Im using version=10.1.1

Im trying to start the containers with a dashboard but i cant mange to do it, i can set a datasource by putting it in /etc/grafana/provisioning/datasources

im mountin the dashboards as a ConfigMap, which has a json file and im setting in various paths, my manifest is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace : default  
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
        - name: grafana
          image: grafana/grafana:latest
          ports:
            - containerPort: 3000
          env:
            - name: GF_INSTALL_PLUGINS
              value: grafana-clock-panel,grafana-simple-json-datasource
          volumeMounts:
            - name: grafana-storage
              mountPath: /var/lib/grafana
            - name: datasource-config
              mountPath: /etc/grafana/provisioning/datasources  # Directorio de configuración automática de datasources
            - name: dashboard-config  
              mountPath: /var/lib/grafana/dashboards/ # Directorio de configuración automática de dashboards
          command: ["grafana-server"]
#        - name: grafana-sc-dashboard
#          image: quay.io/kiwigrid/k8s-sidecar:1.19.2
#          env:
#            - name: LABEL
#              value: "grafana_dashboard"
#            - name: FOLDER
#              value: "/tmp/dashboards"
      volumes:
        - name: grafana-storage
          emptyDir: {}
        - name: datasource-config
          configMap:
            name: grafana-datasource-config  
        - name: dashboard-config
          configMap:
            name: grafana-dashboard-config
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: default  
spec:
  selector:
    app: grafana
  ports:
    - protocol: TCP
      port: 3000
      targetPort: 3000
  type: NodePort

i have tried paths putting the dashboards as:
/etc/grafana/provisioning/dashboards
/var/lib/grafana/dashboards
/etc/grafana/dashboards
but none seem to work

and the configMap:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    grafana_dashboard: "1"
  name: grafana-dashboard-config
data:
  prometheus-dashboard.json: |  # Nombre del archivo debe coincidir con el que se monta en el ConfigMap
    {
      "__inputs": [
        {
          "name": "DS_PROMETHEUS",
          "label": "Prometheus",
          "description": "",
          "type": "datasource",
          "pluginId": "prometheus",
          "pluginName": "Prometheus"
        }
      ],
      "__elements": {},
      "__requires": [
        {
          "type": "grafana",
          "id": "grafana",
          "name": "Grafana",
          "version": "10.1.0"
        },
        {
          "type": "datasource",
          "id": "prometheus",
          "name": "Prometheus",
          "version": "1.0.0"
        },
        {
          "type": "panel",
          "id": "timeseries",
          "name": "Time series",
          "version": ""
        }
      ],
      "annotations": {
        "list": [
          {
            "builtIn": 1,
            "datasource": {
              "type": "grafana",
              "uid": "-- Grafana --"
            },
            "enable": true,
            "hide": true,
            "iconColor": "rgba(0, 211, 255, 1)",
            "name": "Annotations & Alerts",
            "type": "dashboard"
          }
        ]
      },
      "editable": true,
      "fiscalYearStartMonth": 0,
      "graphTooltip": 0,
      "id": null,
      "links": [],
      "liveNow": false,
      "panels": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "fieldConfig": {
            "defaults": {
              "color": {
                "mode": "palette-classic"
              },
              "custom": {
                "axisCenteredZero": false,
                "axisColorMode": "text",
                "axisLabel": "",
                "axisPlacement": "auto",
                "barAlignment": 0,
                "drawStyle": "line",
                "fillOpacity": 0,
                "gradientMode": "none",
                "hideFrom": {
                  "legend": false,
                  "tooltip": false,
                  "viz": false
                },
                "insertNulls": false,
                "lineInterpolation": "linear",
                "lineWidth": 1,
                "pointSize": 5,
                "scaleDistribution": {
                  "type": "linear"
                },
                "showPoints": "auto",
                "spanNulls": false,
                "stacking": {
                  "group": "A",
                  "mode": "none"
                },
                "thresholdsStyle": {
                  "mode": "off"
                }
              },
              "mappings": [],
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "green",
                    "value": null
                  },
                  {
                    "color": "red",
                    "value": 80
                  }
                ]
              }
            },
            "overrides": []
          },
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 0
          },
          "id": 1,
          "options": {
            "legend": {
              "calcs": [],
              "displayMode": "list",
              "placement": "bottom",
              "showLegend": true
            },
            "tooltip": {
              "mode": "single",
              "sort": "none"
            }
          },
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "${DS_PROMETHEUS}"
              },
              "disableTextWrap": false,
              "editorMode": "builder",
              "expr": "calls_answered_total",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "__auto",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "title": "Panel Title",
          "type": "timeseries"
        }
      ],
      "refresh": "",
      "schemaVersion": 38,
      "style": "dark",
      "tags": [],
      "templating": {
        "list": []
      },
      "time": {
        "from": "now-6h",
        "to": "now"
      },
      "timepicker": {},
      "timezone": "",
      "title": "prometheusds",
      "uid": "d06a116f-3b68-4cd6-849c-61c76d45562b",
      "version": 1,
      "weekStart": ""
    }

the dashboards doesnt appear

this are the start and the end of the logs if they are of use:

Deprecation warning: The standalone 'grafana-server' program is deprecated and will be removed in the future. Please update all uses of 'grafana-server' to 'grafana server'
logger=settings t=2023-09-04T18:11:09.252960235Z level=info msg="Starting Grafana" version=10.1.1 commit=0cfa76b22d branch=HEAD compiled=2023-08-31T17:47:57Z
logger=settings t=2023-09-04T18:11:09.253175672Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2023-09-04T18:11:09.253196984Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
logger=settings t=2023-09-04T18:11:09.253214823Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
logger=settings t=2023-09-04T18:11:09.253227114Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
logger=settings t=2023-09-04T18:11:09.253241176Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
logger=settings t=2023-09-04T18:11:09.253256165Z level=info msg=Target target=[all]
logger=settings t=2023-09-04T18:11:09.253273498Z level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2023-09-04T18:11:09.253285906Z level=info msg="Path Data" path=/var/lib/grafana
logger=settings t=2023-09-04T18:11:09.253297548Z level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2023-09-04T18:11:09.253309096Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2023-09-04T18:11:09.253320974Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2023-09-04T18:11:09.253332179Z level=info msg="App mode production"
logger=sqlstore t=2023-09-04T18:11:09.253568747Z level=info msg="Connecting to DB" dbtype=sqlite3
logger=sqlstore t=2023-09-04T18:11:09.253588904Z level=info msg="Creating SQLite database file" path=/var/lib/grafana/grafana.db



logger=secrets t=2023-09-04T18:11:11.212550559Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
logger=local.finder t=2023-09-04T18:11:11.292242172Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled
logger=local.finder t=2023-09-04T18:11:11.292280996Z level=warn msg="Skipping finding plugins as directory does not exist" path=/var/lib/grafana/plugins
logger=query_data t=2023-09-04T18:11:11.295890007Z level=info msg="Query Service initialization"
logger=live.push_http t=2023-09-04T18:11:11.298431883Z level=info msg="Live Push Gateway initialization"
logger=infra.usagestats.collector t=2023-09-04T18:11:14.740739417Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2
logger=modules t=2023-09-04T18:11:14.740950587Z level=info msg=initialising module=secret-migrator
logger=modules t=2023-09-04T18:11:14.740945623Z level=info msg=initialising module=http-server
logger=http.server t=2023-09-04T18:11:14.744230483Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl= socket=
logger=modules t=2023-09-04T18:11:14.747429814Z level=info msg=initialising module=provisioning
logger=provisioning.datasources t=2023-09-04T18:11:14.748752362Z level=info msg="inserting datasource from configuration " name=Prometheus uid=PBFA97CFB590B2093
logger=provisioning.alerting t=2023-09-04T18:11:14.813304124Z level=info msg="starting to provision alerting"
logger=provisioning.alerting t=2023-09-04T18:11:14.813346489Z level=info msg="finished to provision alerting"
logger=modules t=2023-09-04T18:11:14.813396337Z level=info msg=initialising module=background-services
logger=modules t=2023-09-04T18:11:14.813645385Z level=info msg="All modules healthy" modules="[http-server secret-migrator provisioning background-services]"
logger=grafanaStorageLogger t=2023-09-04T18:11:14.813700123Z level=info msg="storage starting"
logger=ngalert.state.manager t=2023-09-04T18:11:14.813733544Z level=info msg="Warming state cache for startup"
logger=ngalert.state.manager t=2023-09-04T18:11:14.848984036Z level=info msg="State cache has been initialized" states=0 duration=35.249022ms
logger=ngalert.multiorg.alertmanager t=2023-09-04T18:11:14.849050187Z level=info msg="Starting MultiOrg Alertmanager"
logger=ngalert.scheduler t=2023-09-04T18:11:14.849046154Z level=info msg="Starting scheduler" tickInterval=10s
logger=ticker t=2023-09-04T18:11:14.849119843Z level=info msg=starting first_tick=2023-09-04T18:11:20Z
logger=grafana.update.checker t=2023-09-04T18:11:14.897584319Z level=info msg="Update check succeeded" duration=83.938116ms
logger=plugins.update.checker t=2023-09-04T18:11:15.084617647Z level=info msg="Update check succeeded" duration=271.075462ms
logger=authn.service t=2023-09-04T18:11:43.244339181Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-09-04T18:11:43.244402603Z level=warn msg=Unauthorized error="user token not found" remote_addr=127.0.0.1 traceID=
logger=context userId=0 orgId=0 uname= t=2023-09-04T18:11:43.244442889Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=127.0.0.1 time_ms=0 duration=525.341µs size=40 referer= handler=/api/live/ws
logger=authn.service t=2023-09-04T18:11:45.603576321Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-09-04T18:11:45.603639059Z level=warn msg=Unauthorized error="user token not found" remote_addr=127.0.0.1 traceID=
logger=context userId=0 orgId=0 uname= t=2023-09-04T18:11:45.603685894Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=127.0.0.1 time_ms=0 duration=609.996µs size=40 referer= handler=/api/live/ws
logger=authn.service t=2023-09-04T18:11:47.878941021Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-09-04T18:11:47.87902324Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=127.0.0.1 time_ms=0 duration=545.069µs size=29 referer=http://localhost:3000/login handler=/
logger=authn.service t=2023-09-04T18:11:47.881146539Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=authn.service t=2023-09-04T18:11:51.156162711Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=1 orgId=1 uname=admin t=2023-09-04T18:11:53.935837553Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=127.0.0.1 time_ms=1 duration=1.712808ms size=0 referer= handler=/api/live/ws
logger=infra.usagestats t=2023-09-04T18:12:31.817108831Z level=info msg="Usage stats are ready to report"

welcome @ianszejer

I am not really familiar with k8 env, but here is something you might want to try. In your manifest, try to follow the below format. I noticed yours is slightly different from this template

          volumeMounts:
            - mountPath: /etc/grafana/provisioning/dashboards
              name: grafana-dashboard
              readOnly: false

good luck and let us know