400 (Bad request) error in Grafana Alloy log deployed in EKS Cluster

Grafana Alloy
1

Hi Team,

I followed the steps as per the documentation, Deploy Grafana on Kubernetes | Grafana documentation using the helm charts.

I exposed Pyroscope service at nodeport so to be accessible from browser. I am able to view Pyroscope dasboard, showing scape metrics of Pyroscope pod and Pyroscope-alloy pod.
However I do not see my-applicaiton-service metric info being pulled.

I also see below 400 error being logged in pyroscope-alloy-0 container pod.

sudo kubectl logs -f pyroscope-alloy-0 -n pyroscope-test

ts=2024-08-30T23:19:42.469250847Z level=error msg=“fetch profile failed” component_path=/ component_id=pyroscope.scrape.pyroscope_scrape_goroutine scrape_pool=pyroscope.scrape.pyroscope_scrape_goroutine target=“{app="dev1-testservice", container="ev1-testservice", instance="10.135.18.251:8080", namespace="default", pod="dev1-testservice-8647b86976-z4wgt", pod_template_hash="8647b86976", service_name="default/dev1-testservice"}” err=“server returned HTTP status (400) Bad Request\r\nThis combination of host and port requires TLS.”

In my service I used below annotation within my service deployment manifest file…

acct-docker.yaml

    annotations:
    kubectl.kubernetes.io/default-container: dev1-testservice
    profiles.grafana.com/memory.scrape: "true"
    profiles.grafana.com/memory.port: "8080"
    profiles.grafana.com/memory.scheme: "http"
    profiles.grafana.com/cpu.scrape: "true"
    profiles.grafana.com/cpu.port: "8080"
    profiles.grafana.com/cpu.scheme: "http"
    profiles.grafana.com/goroutine.scrape: "true"
    profiles.grafana.com/goroutine.port: "8080"
    profiles.grafana.com/goroutine.scheme: "http"

    Env: 
       - name: PYROSCOPE_SERVER_ADDRESS
      value: "http://pyroscope.pyroscope-test.svc.cluster.local.:4040"
    - name: PYROSCOPE_APPLICATION_NAME
      value: "dev1-testservice"

acct-service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: dev1-testservice
name: dev1-testservice
namespace: default
spec:
ports:

  • port: 8080
    nodePort: 32036
    protocol: TCP
    targetPort: 8080
    selector:
    app: testservice
    sessionAffinity: None
    type: NodePort

Appreciate your time and help, so we can complete our demo with mgmt to onboard this tool within our Enterprise Grafana Dashboard.

Thanks,
Suraj