Can Grafana parse/scrape information from a Webpage?

:roll_eyes: After all this - I hit the “refresh roadblock”. I falsely assumed that one could refresh the panels at individual rates. Some at every 10seconds, others - every 5 minutes. While my dashboard contains a weather radar map as well as my new json temperature gauges - I can not set unique refresh rates for the panels. Either I updated the weather radar map every 10 seconds or update the temperature gauges every 5 minutes. Major buzzkill.

With that being said - I learned a lot, so it was not a complete waste of time. Thank you for all of your assist with this, Grant :+1:

1 Like

Totally happy to help if I can.

1 Like

Hello Grant2,
I need a little help with the installation, pls.

  1. Clone the plugin to your Grafana plugins directory.
  2. Build the plugin by running yarn install and then yarn build .

1.) I have done.
But I don´t know exactly how to implement yarn install and yarn build at my raspberry 4. Could yyou pls help me? Wich folder have I to choose? Which is the correct final commands of yarn install and build?

I have to install yarn too I guess. Its not on my machine. I think I will this get done.

Thank you

@jossie Maybe someone else here can help guide you through it. Yarn, mage, etc. are all things which I do not know anything about.

Hi!

This look interesting. I want to scrape a single value from my webpage and get a graph out of it. On my webpage there will only be value.

I am a beginner of this.
Do you now a good step by step tutorial how to do it?

Welcome @tottemannen72

What is the webpage and can you open it with Chrome developer tools to identify the structure of the data and where your value is? It may be quite simple or require some scraping tools.

I would go with python script with beautifulsoup being run by telegrad to push data to your desired tsdb

I have done a scrape script in python.

import requests
from bs4 import BeautifulSoup
URL = "http://ludvikasegel.com/wx/cloudbase.asp"
page = requests.get(URL)
soup = BeautifulSoup(page.content, "html.parser")
molnbas = soup.find_all("div", class_="cloudbase")
print (molnbas[0].text.strip())

Now I want to store the data in a database, and connect it to get a graph in grafana.
That will be the next step, but I dont really know how to do it. But I search the internet, or do you have any suggestions how to proceed?

1 Like

Which database technology and which part of the data that you scrapped. You are asking about non grafana stuff

I use SQLite database.

import requests
import sqlite3
import datetime 

from bs4 import BeautifulSoup
URL = "http://ludvikasegel.com/wx/cloudbase.asp"
page = requests.get(URL)
soup = BeautifulSoup(page.content, "html.parser")
molnbas = soup.find_all("div", class_="cloudbase")
print (molnbas[0].text.strip())


with sqlite3.connect('database.db') as con, open("d:\Cloudbase python\schema.sql") as f:
    con.executescript(f.read())


    timestamp = datetime.datetime.utcnow().isoformat()  # timestamp, change to whatever you want
    desired_value = molnbas[0].text.strip()
    

with sqlite3.connect("d:\Cloudbase python\database.db") as sqlite_connection:
    sqlite_connection.execute(
        "INSERT INTO table_name VALUES (?, ?)",
        (timestamp, desired_value)
    )

I don´t know if I am on right track here, but is it possible to connect to Grafana with this?

1 Like

Are you asking if you can connect grafana to your sqlite db?

Are you asking if you can connect grafana to your sqlite db?

Yes, that is one question.

Check this out

Ok, so a have download grafana server on my Raspberry Pi, installed the SQLite plugin.


My python script runs every 5 minutes and send data to my database.

Now my question is how to get a graph in Grafana and show the graph on my webpage.

Might want to start by reading documentation.

and this

Try things out and then when you are stuck ping back here.

Hi!
I have tried some things out. I got a bar chart anyway, with no time.

But I want a Time Series, and data is missing.
I have the time stamp


I need to convert it to unix timestamp / unix epoch. But I dont know how to do it in the query.
I have checked forum, but dont know how to start.

Go to transformation tab and use the convert field option to convert that column to time

Is that column UTC?

So a manage to get a graph in Time series using convert field.


Time is local time.

To get the graph on my webpage I assume that I need to use port forward, or is it some other workaround for that?

1 Like