Data scraping of a tps database

Good day all. Thank you for reading this. I would like to apologies for the lack of wrong category placement i will be learning from the feedback of the community.

I would like to ask you all for some help please. I only succeeded recently on installing Grafana and Prometheus database to work properly with my Mikrotik. I was Not familiar with CLI and .YML files. so as i learned for the past 2 years i finally got it working.

I would like to ask you all for some on how to make my project a reality with Grafana. As my project is linked to my work it is a bit sensitive.

My server that i work with currently has the following Database where all my info is stored and backed up. The database is IP Driver (IPDS) the extension it woks on is .TPS.

I would like to know what data base do i have to create so that i can make it work with Grafana.

I would like some data to be scraped with in the .TPS database as my data is time sensitive. Would it be posable to create a dashboard that has certain aspects of the info i would like to display.

Thank you

postgres or mysql are pretty easy to connect to with Grafana

please share documentation of this ipds database

Good day fellow members. I have made a flow chart as to what i would like to accomplished. I will submit some details of the IPDS also. I do not have a lot of info to it as i do not have a license myself. My supplier is renting the database on a monthly fee.

Here is IP Driver and Data Server (IPDS) – SoftVelocity.

Clarion Topspeed IPDS.

the extention is .TPS

I hope this is A bit better picture as what i would like to accomplish.

1 Like

if supplier provides you with a json file, then what? will you use raw json to do your grafana visualization (which is doable but tedious and manual)?

other options.

Ask supplier to push the needed data into a database instead of a json file.

1 Like

looks ancient! wonder what the source of the data is. are there devices that push data to it.

You are correct. As i also did my research, this top speed data base is only 16bit and not 32bit yet. Well i would like to get it in a different database but the way they wrote the program it is not possible to convert the database overnight. (corporate greed from Top speed) they waited till they had a lot of clients and locked there database into a proprietary code. so to rewrite all of that well is already in the works for the past 2 years. Just a single license for the data base is already at $300 that R5490,44 per month if i convert from USD to Zar as of 15/05/2024 17:45. I was just wondering if any one else had this Database and a plugin for it.

I will wait till my supplier gets a way around the IPDS data base.

  1. are you trying to also see historical data or from today onward?
  2. what is the source of this data? is it devices pushing data to the tps database? does this tps db come bundled with these devices? what kind of data is it, human readable format? what is the content of the data, what data points are being collected from device A or …

you did mention it is sensitive but give us a general idea.

Good evening

Lets see if i can answer this in the point you are asking.

  1. The way i would like to see this info is max for 14 days. You can say some historical data( maybe if it works well long term)

2.1 what is the source of this data?
yes the info is received from a base station that is connected to Mobile network 3G,4G,5G
and WIFI.

2.2 is it devices pushing data to the tps database?
As my devices
received signals it pass them to my pc via serial (com1 to com8) I have a lot of devices
connected.

2.3 what kind of data is it, human readable format?
The data is received in RAW data see example
Contact Id Codes | PDF | Security Alarm | Equipment

2.4 Does this tps db come bundled with these devices?
No, the device i have connected to my server is not bundled to my software.

2.5 is the content of the data, what data points are being collected from device A or …
I have a DB of clients (Radios) i monitor security alarm systems Bv.
Arming
Disarming
Burglary
Panic
AC Fail

I hope this is the best i can explain the project i want to build.

1 Like

2.2 when you say my devices is this separate than base station

I have in total 8 base stations. al of them are connected to my server via a 9 pin serial cable. Each base station on my server has an encoder to take the raw code and push it to the stack. from the stack it gets entered in to the data base.

confusing. earlier you said your bases were sending data via mobile network,

please draw it out clearly like below.

https://intelligent-monitoring.com/intelligent-monitoring/

No i ment it is reciving signels from 3G,4G 5G and wifi. it sends RAW CID data to my server via a 9 pin serial cable (com1- com8) 8 com ports.

who is the it, that is receiving signal? draw it out.

what protocol/port is this data being sent

1 Like

I hope you can see the names.

1 Like

what is Stack ? who owns it and is it a Clarion product? how is data pushed to Clario backend? what protocol, what port etc

The stack is the Main process that puts all the Raw CID From the Encoders into the database. you can say the stack is the DB. if the stack stops to work you can not open the administration window nor lookup any events.

the stack is not a product of clarion. Intelligent monitoring has been written around clarion the port is not known as the Dev does not share any info on access. all i gathered from the Techs so on it works on an IP base but i am not sure.