Hello Team,
I installed Loki by helm chart: Install Grafana Loki with Helm | Grafana Loki documentation
It works fine, I can see logs in Grafana for my pods etc.
Then I installed Gitlab Runner (kubernetes exectutor) to my cluster: Kubernetes executor | GitLab
Kubernetes exec. contains the main pod called Runner which creates more pods to run CI jobs concurently.
The thing is I’m able to see Runner’s logs (gitlab-runner-gitlab-runner-65c554859c-9hdwl), but not able to see anything from CI pods (runner-gfqp7sqq-project-22859791-concurrent-1qvxq8) – there are not data that Loki can reflect by pod label although pod label exists.
First thing to check might be the output of the CI pods directly:
kubectl -n gitlab-apps logs -f runner-gfqp7sqq-project-22859791-concurrent-1qvxq8
Yes, let me shortly explain how Gitlab CI pod works.
For instance, our Gitlab pipeline contains services and main container (:
maven:acceptance:test:
image: maven:3.6.3-jdk-11 # main
services: # services
- name: $CI_DOCKER_REGISTRY_URL/payment-gateway-service:$NEW_VERSION
- name: $CI_DOCKER_REGISTRY_URL/emulator-service:latest
- name: $CI_DOCKER_REGISTRY_URL/authentication-service:latest
- name: $CI_DOCKER_REGISTRY_URL/nginx-acceptance-tests:latest
- name: $CI_DOCKER_REGISTRY_URL/oracle-db-seeded:latest
alias: oracle
- name: selenium/standalone-chrome
alias: selenium
- name: redis
stage: test:acceptance
script:
- mvn $MAVEN_CLI_OPTS verify
Each of them creates container. I’m able to see logs via kubectl logs command.
Probably something wrong with default scrape config, I thought scrape config would get around me 
I attached all configuration files.
promtail_config: http://pastie.org/p/4x4RVxuyrkRJWLwJ2G9fb2
pod_labels_and_metadata: http://pastie.org/p/5vEkDviEd6Pg6FOjU3wYHt
typical_container_in_pod: Pastie
