Error in setting Prometheus datasource in Grafana on Kubernetes

Grafana
1

I am setting up Grafana on Kubernetes with some custom dashboards and Prometheus as default data source.

All services are running behind Traefik. Prometheus is running at https://myapp.dev/svc/prometheus/ and Grafana is running at https://myapp.dev/svc/grafana/

I can login to both grafana(also able to see dashboards that were added to config) and prometheus, however when I test the Prometheus datasource, I see the following error in grafana logs:

enter image description here

lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=xxx path=/api/datasources/proxy/1/api/v1/query remote_addr=192.168.64.1 referer=https://myapp.dev/svc/grafana/datasources/edit/1/ error="http: proxy error: dial tcp: lookup prometheus on 10.96.0.10:53: no such host"

Grafana deployment

kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: grafana-deployment
  labels:
    app: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
        - containerPort: 3000
        volumeMounts:
          - mountPath: /etc/grafana/provisioning/datasources/
            name: datasource-volume
          - mountPath: /etc/grafana/provisioning/dashboards/
            name: dashboard-volume
          - mountPath: /var/lib/grafana
            name: grafana-storage
      volumes:
        - name: grafana-storage
          emptyDir: {}
        - configMap:
            defaultMode: 420
            name: grafana-datasource-cm
          name: datasource-volume
        - configMap:
            defaultMode: 420
            name: grafana-dashboard-cm
          name: dashboard-volume

Configmap

kind: ConfigMap
apiVersion: v1
metadata:
  name: grafana-datasource-cm
data:
  datasource.yml: |-
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      orgId: 1
      access: proxy
      url: http://prometheus:9090/svc/prometheus/
      basicAuth: false

I am not sure on how to fix this error and test Prometheus datasource.