Is it possible and if yes, how to configure Grafana for this?
Yes, there are many ways to monitor these databases.
First you need a collector to gather metrics about your database and store those metrics in a time series database. Then you can visualize and alert on this metrics in Grafana.
Here are a couple of ready made dashboards for mysql. You just need to install a metric collector (for example node exporter for Prometheus)