Im gettting below error in grafana can anyone help me out
logger=dashboard-service t=2026-09-18T00:00:25.163766987+06:30 level=error msg="Failed to cleanup k8s dashboard resources" error="org 1: failed to get last resource version: unsupported time format 2026-09-16 16:29:28.897727688 +0630 +0630 m=+170852.557746742: parsing time "2026-09-16 16:29:28.897727688 +0630 +0630 m=+170852.557746742" as "2006-01-02 15:04:05.9999999 Z07:00": cannot parse "+0630 +0630 m=+170852.557746742" as "Z07:00""
logger=ngalert.multiorg.alertmanager t=2026-09-18T00:00:25.944522587+06:30 level=error msg="Unable to create Alertmanager for org" org=1 error="error reading file 'notifications' from database: unsupported time format 2026-09-08 13:18:43.043512204 +0630 +0630 m=+351905.712576920: parsing time "2026-09-08 13:18:43.043512204 +0630 +0630 m=+351905.712576920" as "2006-01-02 15:04:05.9999999 Z07:00": cannot parse "+0630 +0630 m=+351905.712576920" as "Z07:00""
logger=ngalert.sender.router rule_uid=ffx7jkost5ddsf org_id=1 t=2026-09-18T00:00:26.735154256+06:30 level=info msg="Sending alerts to local notifier" count=1
logger=ngalert.sender.router rule_uid=ffx7jkost5ddsf org_id=1 t=2026-09-18T00:00:26.735198964+06:30 level=error msg="No external or internal notifier - alerts not delivered" count=1
logger=ngalert.sender.router rule_uid=bfx7jua6c8em8d org_id=1 t=2026-09-18T00:00:30.024407281+06:30 level=info msg="Sending alerts to local notifier" count=13
logger=ngalert.sender.router rule_uid=bfx7jua6c8em8d org_id=1 t=2026-09-18T00:00:30.024446592+06:30 level=error msg="No external or internal notifier - alerts not delivered" count=13
logger=ngalert.sender.router rule_uid=efx7l2q4kedj4d org_id=1 t=2026-09-18T00:00:37.515243518+06:30 level=info msg="Sending alerts to local notifier" count=1
logger=ngalert.sender.router rule_uid=efx7l2q4kedj4d org_id=1 t=2026-09-18T00:00:37.515319637+06:30 level=error msg="No external or internal notifier - alerts not delivered" count=1
logger=ngalert.sender.router rule_uid=ffx7ksmlyd1q8b org_id=1 t=2026-09-18T00:00:46.742581079+06:30 level=info msg="Sending alerts to local notifier" count=1
logger=ngalert.sender.router rule_uid=ffx7ksmlyd1q8b org_id=1 t=2026-09-18T00:00:46.742631808+06:30 level=error msg="No external or internal notifier - alerts not delivered" count=1
logger=dashboard-service t=2026-09-18T00:00:55.164278687+06:30 level=error msg="Failed to cleanup k8s dashboard resources" error="org 1: failed to get last resource version: unsupported time format 2026-09-16 16:29:28.897727688 +0630 +0630 m=+170852.557746742: parsing time "2026-09-16 16:29:28.897727688 +0630 +0630 m=+170852.557746742" as "2006-01-02 15:04:05.9999999 Z07:00": cannot parse "+0630 +0630 m=+170852.557746742" as "Z07:00""
The first error is the main issue. The timestamp
2026-09-16 16:29:28.897727688 +0630 +0630 m=+170852.557746742
contains Go monotonic-clock information (m=+...) and a duplicated timezone offset, which Grafana cannot parse. This is causing the dashboard-service cleanup and Alertmanager notifications read to fail.
This matches
opened 10:22AM - 27 Jul 26 UTC
closed 09:27AM - 10 Aug 26 UTC
type/bug
area/backend/db
area/backend/db/sqlite
area/backend/db/migration
type/regression
automated-triage
### What happened?
After upgrading Grafana from 12.x to 13.1.0 using the same p… ersistent SQLite database, Grafana fails to start.
The server initializes successfully, loads plugins, provisioning, and the HTTP server, but then exits during service startup with the following error:
```
unsupported time format
parsing time "2026-07-26 11:47:47.112233638 +0330 +0330 m=+84630.747193824"
```
Later the process terminates with:
```
invalid service state: Failed
expected: Terminated
```
Downgrading back to Grafana 12.x using the same database starts successfully without any changes.
From the logs, it appears that Grafana 13.x cannot parse a persisted timestamp containing Go monotonic clock information (`m=+...`).
### What did you expect to happen?
Grafana should successfully migrate and start after upgrading from Grafana 12.x to 13.1.0 using an existing SQLite database, without requiring manual database modifications.
### Did this work before?
Yes.
The same database works correctly on Grafana 12.x.
The issue only appears immediately after upgrading to Grafana 13.1.0.
### How do we reproduce it?
1. Start Grafana 12.x using SQLite as the backend database.
2. Configure Grafana normally (dashboards, users, etc.).
3. Stop Grafana.
4. Upgrade the Docker image to Grafana 13.1.0 while keeping the existing SQLite database.
5. Start Grafana.
Expected:
Grafana starts normally after performing any required migrations.
Actual:
Grafana exits during startup with an "unsupported time format" error while reading persisted state.
### Is the bug inside a dashboard panel?
### Relevant log
```
unsupported time format
parsing time "2026-07-26 11:47:47.112233638 +0330 +0330 m=+84630.747193824"
invalid service state: Failed
expected: Terminated
```
The timestamp contains Go monotonic clock information (`m=+...`), which appears to be persisted and later parsed during startup.
This may be a regression in the migration or KV store timestamp handling introduced in Grafana 13.x.
### Environment (with versions)?
Grafana: 13.1.0
Previous version:
12.x
Database:
SQLite
Deployment:
Docker
OS:
Ubuntu 24.04 LTS
Browser:
N/A (Grafana never becomes available)
### Grafana platform?
Docker
### Datasource(s)?
Not datasource specific. Grafana exits during startup before any datasource is initialized.
which reports the same unsupported time format problem when upgrading from Grafana 12.x to 13.1.0 with SQLite.
main ← maithilijoshi20:fix/sqlite-legacy-time-parser
opened 06:22AM - 03 Aug 26 UTC
**What is this feature?**
Adds backward-compatible parsing for malformed SQLite… timestamps previously serialized by modernc as Go time.Time.String values.
**Why do we need this feature?**
Grafana now writes stable SQLite time values via _time_format=sqlite, but existing databases can contain legacy values with a duplicated numeric offset and monotonic clock suffix. Those values prevent Grafana from starting after an upgrade.
**Who is this feature for?**
Grafana administrators upgrading an existing SQLite-backed installation.
**Which issue(s) does this PR fix?**
Fixes #129295
**Special notes for your reviewer:**
- Preserves existing timestamp formats and adds the legacy parsing only as a fallback.
- Includes regression coverage for the exact reported timestamp and the no-monotonic-suffix variant.
- Verified with:
- `go test ./pkg/util/xorm -count=1`
- `go test ./pkg/util/sqlite -run TestStableTimeFormat -count=1`
main ← fix-129518
opened 10:08AM - 29 Jul 26 UTC
The `modernc.org/sqlite` driver serializes `time.Time` values with `time.Time.St… ring()` when no `_time_format` is set. Go documents that as a debugging representation: it can embed a monotonic reading or a duplicated numeric zone offset, producing values that don't round-trip.
This defaults the DSN wrapper to `_time_format=sqlite` so times are stored in a stable format. Explicit user configuration still takes precedence, and the default now applies even when the DSN has no query parameters.
Fixes #129518
The No external or internal notifier - alerts not delivered messages appear to be a consequence of the Alertmanager initialization failure shown immediately before them.
Please provide your Grafana version, database type, and whether this started after an upgrade, particularly if you upgraded from Grafana 12.x to 13.x.
Hello ,
Please find below the grafana details, since its a new setup and not an upgrade I have deleted the db and started the configuration from fresh, but facing similar error in logs for unsupported time format,
Database type: sqllite(default)
curl -s http://127.0.0.1:3000/api/health
{
"database": "ok",
"version": "12.4.4",
**But can see similar error
**error reading file ‘notifications’ from database: unsupported time format 2026-09-21 13:30:49.873737007 +0630 +0630 m=+18902.869622070: parsing time “2026-09-21 13:30:49.873737007 +0630 +0630 m=+18902.869622070” as “2006-01-02 15:04:05.9999999 Z07:00”: cannot parse “+0630 +0630 m=+18902.869622070” as “Z07:00"” No external or internal notifier - alerts not delivered ” count=1
logger=app-registry t=2026-09-21T08:15:52.738768735+06:30 level=info msg="app registry initialized"
logger=backgroundsvcs.managerAdapter t=2026-09-21T08:15:52.738787091+06:30 level=info msg=starting module=\*live.GrafanaLive
logger=backgroundsvcs.managerAdapter t=2026-09-21T08:15:52.738823071+06:30 level=info msg=starting module=\*dualwrite.ZanzanaReconciler
logger=backgroundsvcs.managerAdapter t=2026-09-21T08:15:52.7388488+06:30 level=info msg=starting module=\*notifications.NotificationService
logger=backgroundsvcs.managerAdapter t=2026-09-21T08:15:52.738873438+06:30 level=info msg=starting module=\*remotecache.RemoteCache
logger=backgroundsvcs.managerAdapter t=2026-09-21T08:15:52.738900021+06:30 level=info msg=starting module=\*service.DashboardUpdater
logger=backgroundsvcs.managerAdapter t=2026-09-21T08:15:52.739390704+06:30 level=info msg=starting module=\*metric.Service
logger=backgroundsvcs.managerAdapter t=2026-09-21T08:15:52.739676244+06:30 level=info msg=starting module=\*manager.SecretsService
logger=backgroundsvcs.managerAdapter t=2026-09-21T08:15:52.739721867+06:30 level=info msg=starting module=\*dynamic.KeyRetriever
logger=backgroundsvcs.managerAdapter t=2026-09-21T08:15:52.740548245+06:30 level=info msg=starting module=\*anonimpl.AnonDeviceService
logger=backgroundsvcs.managerAdapter t=2026-09-21T08:15:52.742806491+06:30 level=info msg="All modules healthy"
logger=ngalert.state.manager t=2026-09-21T08:15:52.743101673+06:30 level=info msg="State cache has been initialized" states=0 duration=6.67138ms
logger=ngalert.scheduler t=2026-09-21T08:15:52.743141336+06:30 level=info msg="Starting scheduler" tickInterval=10s maxAttempts=3
logger=ngalert.scheduler t=2026-09-21T08:15:52.743198611+06:30 level=info msg=starting component=ticker first_tick=2026-09-21T08:16:00+06:30
logger=http.server t=2026-09-21T08:15:52.763899936+06:30 level=info msg="HTTP Server Listen" address=\[::\]:3000 protocol=http subUrl= socket=
**logger=infra.usagestats t=2026-09-21T08:15:52.819313059+06:30 level=error msg="Failed to get last sent time" error="unsupported time format 2026-09-20 16:45:22.236775652 +0630 +0630 m=+172811.200635360: parsing time "2026-09-20 16:45:22.236775652 +0630 +0630** m=+172811.200635360" as "2006-01-02 15:04:05.9999999 Z07:00": cannot parse "+0630 +0630 m=+172811.200635360" as "Z07:00""
Hi,
Since this is a fresh 12.4.4 install with a new SQLite DB, this is not an upgrade issue, and nothing in your setup looks like the cause.
Grafana stores timestamps in SQLite in Go’s time.Time.String() format (e.g. 2026-09-21 13:30:49.873737007 +0630 +0630 m=+18902.869622070), and Grafana can’t parse that format when it reads the value back. All the errors in your logs share this root cause →
Alertmanager for org 1 fails to start (“Unable to create Alertmanager for org”), which is why you see “No external or internal notifier → alerts not delivered”infra.usagestats: “Failed to get last sent time”dashboard-service: “Failed to cleanup k8s dashboard resources”
Workaround → move Grafana’s database from SQLite to Postgres.
In Postgres:
CREATE USER grafana WITH PASSWORD '<password>';
CREATE DATABASE grafana OWNER grafana;
In grafana.ini:
[database]
type = postgres
host = 127.0.0.1:5432
name = grafana
user = grafana
password = <password>
Restart Grafana:
sudo systemctl restart grafana-server
This starts an empty database, so recreate your data sources and alert rules. The Alertmanager error should then be gone alerts will go to whatever contact points you have configured.
