Grafana loki's helm chart doesn't scrape logs from Gitlab runner (Kubernetes exectutor)

Grafana Loki
#1

Hello Team,

I installed Loki by helm chart: Helm | Grafana Labs
It works fine, I can see logs in Grafana for my pods etc.

Then I installed Gitlab Runner (kubernetes exectutor) to my cluster: The Kubernetes executor | GitLab

Kubernetes exec. contains the main pod called Runner which creates more pods to run CI jobs concurently.

The thing is I’m able to see Runner’s logs (gitlab-runner-gitlab-runner-65c554859c-9hdwl), but not able to see anything from CI pods (runner-gfqp7sqq-project-22859791-concurrent-1qvxq8) – there are not data that Loki can reflect by pod label although pod label exists.

image
image698×98 12.5 KB

#2

First thing to check might be the output of the CI pods directly:

kubectl -n gitlab-apps logs -f runner-gfqp7sqq-project-22859791-concurrent-1qvxq8
#3

Yes, let me shortly explain how Gitlab CI pod works.

For instance, our Gitlab pipeline contains services and main container (:

maven:acceptance:test:
  image: maven:3.6.3-jdk-11 # main
  services: # services
    - name: $CI_DOCKER_REGISTRY_URL/payment-gateway-service:$NEW_VERSION
    - name: $CI_DOCKER_REGISTRY_URL/emulator-service:latest
    - name: $CI_DOCKER_REGISTRY_URL/authentication-service:latest
    - name: $CI_DOCKER_REGISTRY_URL/nginx-acceptance-tests:latest
    - name: $CI_DOCKER_REGISTRY_URL/oracle-db-seeded:latest
      alias: oracle
    - name: selenium/standalone-chrome
      alias: selenium
    - name: redis
  stage: test:acceptance
  script:
    - mvn $MAVEN_CLI_OPTS verify

Each of them creates container. I’m able to see logs via kubectl logs command.
Probably something wrong with default scrape config, I thought scrape config would get around me :slight_smile:

I attached all configuration files.

Peek 2021-02-10 10-34
Peek 2021-02-10 10-341090×709 416 KB

promtail_config: Pastie
pod_labels_and_metadata: Pastie

#4

typical_container_in_pod: Pastie