Hello,
I successfully configured the Grafana Agent to send some logs from one server to a Grafana Cloud instance.
Now I want to configure the same type of setup but on a different internal server to send different logs to the same Grafana cloud instance. I tried to do a similar configuration than the first server for which everything works, but when I generate test error logs to test my setup on my second server, nothing is displayed in the Grafana Cloud instance.
Here are the checks that I made:
- Last version of Grafana Agent installed: v0.36.0
- Grafana Agent service running
- positions file updated with a new line corresponding to the test log file soon after generating it.
- servers are on Windows Server 2019
Here is the agent-config.yaml from the server from which the logs are not sent. I just replaced the username value by “username” and the password value by “APIKey”.
I tested the “Teliway” job, which is not available to select in the Explore section of my Grafana Cloud instance during the timeframe when I generated the test log file, as if no log was sent to Grafana Cloud.
integrations:
prometheus_remote_write:
- basic_auth:
password: APIKey
username: username
url: https://prometheus-prod-22-prod-eu-west-3.grafana.net/api/prom/push
agent:
enabled: true
relabel_configs:
- action: replace
source_labels:
- agent_hostname
target_label: instance
- action: replace
target_label: job
replacement: "integrations/agent-check"
metric_relabel_configs:
- action: keep
regex: (prometheus_target_.*|prometheus_sd_discovered_targets|agent_build.*|agent_wal_samples_appended_total|process_start_time_seconds)
source_labels:
- __name__
# Add here any snippet that belongs to the `integrations` section.
# For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
logs:
configs:
- clients:
- basic_auth:
password: APIKey
username: username
url: https://logs-prod-013.grafana.net/loki/api/v1/push
name: integrations
positions:
# filename: /tmp/positions.yaml
filename: "D:\\Grafana\\Positions\\positions.yaml"
scrape_configs:
# Add here any snippet that belongs to the `logs.configs.scrape_configs` section.
# For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
- job_name: integrations/agent
journal:
max_age: 24h
labels:
instance: <hostname>
job: integrations/agent
pipeline_stages:
- match:
selector: '{unit!="grafana-agent.service"}'
action: drop
drop_counter_reason: only_keep_grafana_agent_logs
- regex:
expression: "(level=(?P<log_level>[\\s]*debug|warn|info|error))"
- labels:
level: log_level
relabel_configs:
- source_labels: ['__journal__systemd_unit']
target_label: 'unit'
- job_name: YoozArchivesErreurs
encoding: ISO-8859-1
static_configs:
- targets:
- localhost
labels:
job: Yooz
level: error
__path__: "C:\\PGI01\\Yooz_Exchange\\Dev\\Archives erreurs\\*.ERR"
- job_name: YoozWORK
encoding: ISO-8859-1
static_configs:
- targets:
- localhost
labels:
job: Yooz
level: error
__path__: "C:\\PGI01\\Yooz_Exchange\\Dev\\WORK\\**\\*.ERR"
- job_name: TeliwayCegid
encoding: ISO-8859-1
static_configs:
- targets:
- localhost
labels:
job: Teliway
level: error
__path__: "C:\\PGI01\\Import_Teliway\\**\\*.ERR"
metrics:
configs:
- name: integrations
remote_write:
- basic_auth:
password: APIKey
username: username
url: https://prometheus-prod-22-prod-eu-west-3.grafana.net/api/prom/push
scrape_configs:
# Add here any snippet that belongs to the `metrics.configs.scrape_configs` section.
# For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
global:
scrape_interval: 60s
# wal_directory: /tmp/grafana-agent-wal
wal_directory: "C:\\ProgramData\\grafana-agent-wal"
Could you please help me on how to debug this?
Thanks