Hi,
can Grafana parse/scrape information from a Webpage? Is there a plugin available?
1 Like
Funny. I just logged in to ask almost that exact question. I have several temperature sensors that constantly advertise the current temp to a text file on port 80 via a python script. I’d like to have grafana cull these into a single dashboard. It’s a piece of cake with plotly-dash but I can’t seem to do it with Grafana / html-type plugins. I keep getting CORS errors.
I do something similar and use NodeRED to scrape the website and send the data to InfluxDB, then use Grafana to display the Influx data.
There are many good tutorials on setting up NodeRED, Influx and Grafana.
2 Likes
Yeah, I do it this way too. Home Server → mariaDB ->Grafana.
But from time to time you need to show a present value without a chart (past values). So it would be nice to fetch this data from a webpage and show it within grafana. For example the price of oil. Advantage: reduce the last of the system.
1 Like
Dittos here [Need for a “current dynamic value” reading + Home Server → mariaDB ->Grafana] - but, on a side note - I find that I do not always get the latest data from the db - even when extending the " last >= now() - interval 4 minute" when I am dumping the HD temp to the DB every minute [and have verified that the values are being recorded]:
SELECT
last AS “time”,
hdd0temp AS “HD1”,
hostname AS “HOST”
FROM host_info
WHERE
hdd0temp != “null”
AND
last >= now() - interval 4 minute
Hence, this is why I am trying to go directly to the server web page source for my temperature readings
Can you share the output of the webpage that you are trying to grab the temperature from? Is it just a single value? If you format it as JSON or CSV, you can use the JSON or CSV datasource plugins to have Grafana read the datapoint, but even then I think it cannot refresh faster than 5 seconds.
If you want true realtime display of your temperature in Grafana, then I recommend using the MQTT datasource plugin and set up an MQTT broker. Depending on your temperature sensors and the equipment you are using, I can help you with that.
Thanks Grant. The output of the web page is simply a text file that outputs text - the current temperature that is updated every 5 seconds. I’m not necessarily looking to get that level of granularity in grafana [although it would be great if I could]. I can run the update every 30 seconds, if need be. But, currently - I have plotly scraping / referencing this text every 5 seconds and it works perfectly.
As I am using a Python script to generate that temp.txt, you are stating that if I output the temperature as json, Grafana can read it? What about CORS restrictions?
I am using DS18B20 1-wire’s to read thermals throughout the house. I hadn’t thought of trying MQTT. I can toy with, this weekend, that as long as you are fairly confident that I can use one single graph to monitor several sensors, using MQTT?
If so, I’d gladly accept your help with it - let me do a brief study on it, first.
@discarn8 I do not know much about Plotly, but from what I can tell, it uses various tools to plot data in pretty formats. It sounds like you are happy with the results there, so what is Grafana offering that makes you want to use that instead of Plotly?
To answer your questions, yes, if your data can be returned as JSON, then this plugin should display it nicely in Grafana. Ditto for CSV using this plugin.
Setting up an MQTT broker is very simple. Think of it like a post office. Various devices (e.g. your temperature sensors) can publish data to the MQTT broker using a topic of your choice (e.g. kitchen) and then other devices (incl. Grafana) can subscribe to the MQTT broker and specify the topic they wish to see.
Right now, I am using both - Grafana and plotly, for my graphs. Personally, I’d like to stick to a single dashboard provider. That and, I love Grafana, thus far. I’m just a little disappointed in a) the inability to grab / scrape a simple text field from a local web page and b) the poor performance of pulling small amounts of data from a mysql database. The db query performance may be my fault, but I’m not sure how. I also discovered that there is a “heatmap” plugin for grafana. I’m not sure if it is the same as / incorporates plotly, but I may look at that, as well and see if I can possibly leverage that.
Thank you for the heads-up and intro to MQTT. It sounds fairly easy - I’ll try to make time to check it out, this weekend.
Just to answer your original question about using Grafana to display in one single graph two different streams, in realtime (this graph “crawls” along every second or so…I have it set to display the last 3 minutes of data).
Perfect - that’s where I am headed.
Ok, I’m exporting the temperature to json and it’s available by browsing to the URL. I then downloaded the json plugin you mentioned, and added it to my data sources [adding the json data source url]. This is the json data I am exporting
{“temp”: “75.2”}
When I try to go in to create a new dashboard and select the json data source, how am i accessing this data?
I’ve tried adding “GET /temp” for my query but I get nothing
First, you need to specify HTTP and input the URL where your JSON data resides:
You could also add a second (third, fourth, etc) JSON API datasource with a different URL to grab, for example, weather data from a URL that returns JSON data.
If you do the above, and click Save & Test, does it work?
If yes, then the panel would look something like this:
Here is a thread where I showed some of my JSON plugin settings:
https://community.grafana.com/t/darksky-alternative/57037
Yes sir - I added the data source almost verbatim … and it test good
Except - I do not appear to have the same options you do, when adding it to the dashboard…
There are several JSON datasource plugins, and it appears the one you installed is not the one that I have used. Can you search for and add this one?
You are absolutely correct, sir - My sincere apologies - updating - standby
BEAUTIFUL! Thank you! Ok. This I can work with. The granularity isn’t too bad, either @ ~5sec
Now… is there any way to add 3 more sources to the one panel or will I need to create 3 separate panels - one for each data source?
1 Like
When you say you want to add 3 more sources to one panel, does that mean you want to view 3 temperatures in 1 gauge? If yes, then I have never seen that done.
If you want multiple gauges in one panel, with each gauge’s value coming from the properly formatted JSON, then I think you could do that. In my example below, I am grabbing temperature and humidity (plus the weather icon) from the same JSON URL
Correct - yes, sir. I want to add 3 more gauges to the panel - but each gauge is sourced by a unique json-data source. The first one was using 192.168.0.2. The next 3 would come from the next 3 consecutive IP sources. I.e.: JSON-SOURCE2=192.168.0.3/jtemp, JSON-SOURCE3=192.168.0.4/jtemp, JSON-SOURCE4=192.168.0.5/jtemp. Then combining these 4 json data sources into one panel to provide 4 different gauges - one for each temperature zone.
I’ll play with it. Grafana is smart. Grafana is capable. Me? Well… I’m learning Grafana. The door is there - I simply must use the right key to unlock it 
1 Like
Hi there,
to subscribe from an mqtt-server is a nice feature. I guess it is this plugin?
How does this work? Any subscription will be received by the mqtt-client of Grafana at any time? Even if no one is logged into the Dashboard of Grafana or nobody is watching an embedded panel at a webpage? Or is the subscription updated only if you are logged in respective you have to wait for the next published information to get the information?
Home Server ->writing into mariaDB → Grafana
will become
Home Server → Grafana. Nice.
What I meant with my initial question was: Fetch information directly from a web page. For example you wanne get the USD/EUR price form yahoo markets. So there is no plugin which can load the webpage html code and filter for the price?
@jossie Yes, that is the MQTT plug in, and it works as you described.
Remember you are only viewing the latest published info, so if you wanted to see the price of oil on Feb. 23, 2022, you could not do that. Of course, this is why you have MariaDB.
I use Node-RED to simultaneously publish data to both Influx (one reading per minute) and publish data to the MQTT broker every 500 milliseconds.
Re: your question about parse/scrape information from a webpage, if you are after the USD/EUR exchange rate, there is probably some URL that offers this info in JSON or CSV format. You would use either the JSON or CSV datasource plugin to get that information. But like MQTT, it would not be stored anywhere.
1 Like