Logs not publishing to Cloud with promtail

Hello,

I have grafana + loki configured locally with docker and working properly. I am trying to send the logs to the Cloud but it doesn’t log anything, no logs are displayed on Grafana Cloud exploring grafanacloud-$project-logs and no labels are found when testing connection. This is the only thing I change in the promtail configuration:

clients:
#  - url: http://loki:3100/loki/api/v1/push
  - url: https://$userId:$password@logs-prod-012.grafana.net/api/prom/push

I have also tried directly with Postman with no luck, even getting a “204 No content” response. It’s a new Trial account from yesterday.

Which can be the issue?

Thanks

Hi @xavideveloper !

The promtail/config.yaml file should contain details about the Promtail server, where positions are stored, how to scrape logs from files, and where to send them. In the example snippet you shared, I only see the clients section.

In your Grafana Cloud account portal, if you click on the Send logs button on the Loki tile for your stack, you can find a populated remote_write with your Loki username (instanceID), Loki password (API Key or Access Token) and remote_write endpoint. Review the “send logs from a standalone host” section on this same page in your account portal for a full config file example for scraping /var/log .

In your hosted Grafana instance, if you navigate to the Connections section you can generate configuration files and there is a guide for this here as well: Collect logs with Promtail

If everything looks fine with the Promtail configuration file, check your system logs to see if there are any other details recorded for why Promtail cannot ship logs to your hosted Loki in Grafana Cloud.

For more details about configuring Promtail, see Configuration | Grafana Loki documentation

And for more info about the Loki Push API, see Promtail | Grafana Loki documentation

An example Promtail configuration file in grafana/loki: loki/clients/cmd/promtail/promtail-local-config.yaml at main · grafana/loki · GitHub

Hi Melody,

thanks for you post. I have checked the documentation you provided, generated another admin API Key but it is still not working. I have looked for “remote_write” on the page grafanacom/orgs/treenation/hosted-logs/$myUserId#sending-logs but there is no such field, I think it’s the “client - url” field.

Where can I check Promtail logs inside docker image “grafana/promtail:2.8.2”? I have looked for them but could not find anything.

As I said before, locally with grafana/loki:2.8.2 and grafana/grafana:latest it works fine, but just changing promtail configuration “client - url” to “https://$userId:$password@logs-prod-012.grafana.net/api/prom/push” doesn’t work.

I have tried adding a Loki Data Source inside my Grafana instance with “https://logs-prod-012.grafana.net:3100” and Basic Auth my userId and API Key but it says “Failed to update datasource”

Thanks

1 Like

Hi @xavier6272 ah I see, you’re correct. That page does not refer to the config snippets for sending logs to the hosted Loki instance details as “remote_write”. The content is provided under the Sending Logs title and shows several different options depending on where Promtail is running.

There are two resources linked here from the Troubleshooting Promtail docs that might help identify where the issue is:

Also, where are the logs located that Promtail should be scraping? I noticed in this Loki documentation that we state:

It’s fairly difficult to tail Docker files on a standalone machine because they are in different locations for every OS. We recommend the Docker logging driver for local Docker installs or Docker Compose.

Regarding adding a Loki Data Source to your self-managed Grafana environment to connect to your hosted Loki instance in Grafana Cloud, you should be able to accomplish this. Use the “Grafana Data Source settings” from the Loki details page and remove the port from the host address. That should allow you to query the hosted Loki data source in your self-managed Grafana but some data will need to be stored there first to be effective.

Hi Melody,

it was a problem with the format of the data sent and Grafana cloud configuration.

Thank you very much for your help.