API endpoint /org/preferences 404's on first use with Terraform

Grafana
,
1

I’m setting up a new cluster with terraform and when I first come to set the org preferences, it gets a 404. All other dashboards, datasources etc, all go in fine, but the trivial org tweaks don’t work. Retrying and it works as expected??

resource “grafana_organization_preferences” “grafana” {
theme = “dark”
timezone = “utc”
week_start = “monday”
home_dashboard_uid = “home”
}

Error: [PUT /org/preferences] updateOrgPreferences (status 404): {}

with grafana_organization_preferences.grafana,
on main.tf line 61, in resource “grafana_organization_preferences” “grafana”:
61: resource “grafana_organization_preferences” “grafana” {

I’ve tried making the preferences a dependency of folders, for example, but it still always fails first time, works second time. Any clues?

2

You didn’t provide any usefull details/logs, so blind guess: grafana is starting and org is not created, when you are trying to uodate org settings, so 404 = org not found.

Then finally (default) org is created, so next try is successful.

3

If I can create every dashboard and alert fine, the default org must surely already exist? I can create a new org fine, but that shows up as org id 2, so again, org id 1 is clearly already there somewhere.

I’m not sure what other logs I could possibly provide, nothing has happened on the system at this point, and /api/org/preferences just should surely never be able to 404 through any valid usage.

4

That’s not a prove that org 1 exists. You can delete org 2 and create new one then - I bet that new org id will be 3 (also when org id 2 is deleted).
Go to DB and check DB table directly.

5

OK that makes sense. I’m not actually looking to create a new one either way though, just set basic stats on the default one. Why would that not work on a new install? From a time perspective, this isn’t being done immediately at the point it’s deployed. 5, 10 minutes later, it still never works first time, but fine on the second one. Is there something significant when deploying alerts, folders or dashboards that is changing something between executions as nothing else is being done to the environment at all.

6

Everything is guess. You didn’t provide reproducible example. Start with standard debugging:

Grafana can provide valuable information about certain issues through its own debug logs. These logs can help troubleshoot and identify the root cause of problems (especially if they are related to anything that can be configured via Grafana config file).

To enable debug logging edit the configuration file grafana.ini:

[log]
# Either "console", "file", "syslog". Default is console and file
# Use space to separate multiple modes, e.g. "console file"
mode = console file

# Either "debug", "info", "warn", "error", "critical", default is "info"
level = debug

Then restart grafana for the setting to go into effect.

Replicate the problem and check Grafana logs. However, before posting the debug logs on the community forum, it is important to sanitize any private details such as passwords, tokens, IPs, names, … By including properly formatted debug logs (with sanitized information) and details about your installations (e.g. exact Grafana version, how it was installed, OS details, …), you greatly enhance the chances of receiving accurate assistance and solutions.

7

Hmm, I should have tried to manually replicate it first, as it states the dashboard is not found. I wouldn’t have thought that would make a 404 appropriate, but apparently by convention it is. So that was the answer - the dashboard hand’t been created yet, which seems very obvious to me now. I’d made it a dependency of folders, but not of dashboards, so I assume that change will cover this off.