Lost data, broken dashboards and our clients' web app iframe mirrors embedding page

Today, we encountered a major issue where all our Grafana dashboards inoperable (aka broken). To stress the significance of this, these dashboards contained several months’ worth of critical data from our pilot customers.

Here’s the flow of our data process:

  1. Our devices transmit telemetry data to our server.
  2. This data is then channeled to an Amazon EC2 instance.
  3. From there, the data is presented on Grafana/Prometheus dashboards. These dashboards are embedded within our web application pages using iFrames.

However, the current situation is as follows:

  • Instead of displaying the intended dashboards, the iFrame within our web application seems to mirror the very page it’s embedded in.
  • Upon accessing the Grafana platform directly, many widgets are devoid of data, except for a few that still display historical data.

We’ve been extensively searching for solutions online, but to no avail. The implications of this situation are disastrous for our company if we can’t rectify the issue and retrieve our client’s data.