Grafana loki's helm chart doesn't scrape logs from Gitlab runner (Kubernetes exectutor)

[thelastemailofmylife]

Hello Team,

I installed Loki by helm chart: Install Grafana Loki with Helm | Grafana Loki documentation
It works fine, I can see logs in Grafana for my pods etc.

Then I installed Gitlab Runner (kubernetes exectutor) to my cluster: Kubernetes executor | GitLab

Kubernetes exec. contains the main pod called Runner which creates more pods to run CI jobs concurently.

The thing is I’m able to see Runner’s logs (gitlab-runner-gitlab-runner-65c554859c-9hdwl), but not able to see anything from CI pods (runner-gfqp7sqq-project-22859791-concurrent-1qvxq8) – there are not data that Loki can reflect by pod label although pod label exists.

[webwurst]

First thing to check might be the output of the CI pods directly:

kubectl -n gitlab-apps logs -f runner-gfqp7sqq-project-22859791-concurrent-1qvxq8

[thelastemailofmylife]

Yes, let me shortly explain how Gitlab CI pod works.

For instance, our Gitlab pipeline contains services and main container (:

maven:acceptance:test:
  image: maven:3.6.3-jdk-11 # main
  services: # services
    - name: $CI_DOCKER_REGISTRY_URL/payment-gateway-service:$NEW_VERSION
    - name: $CI_DOCKER_REGISTRY_URL/emulator-service:latest
    - name: $CI_DOCKER_REGISTRY_URL/authentication-service:latest
    - name: $CI_DOCKER_REGISTRY_URL/nginx-acceptance-tests:latest
    - name: $CI_DOCKER_REGISTRY_URL/oracle-db-seeded:latest
      alias: oracle
    - name: selenium/standalone-chrome
      alias: selenium
    - name: redis
  stage: test:acceptance
  script:
    - mvn $MAVEN_CLI_OPTS verify

Each of them creates container. I’m able to see logs via kubectl logs command.
Probably something wrong with default scrape config, I thought scrape config would get around me

I attached all configuration files.

promtail_config: http://pastie.org/p/4x4RVxuyrkRJWLwJ2G9fb2
pod_labels_and_metadata: http://pastie.org/p/5vEkDviEd6Pg6FOjU3wYHt

[thelastemailofmylife]

typical_container_in_pod: Pastie