I had to change the serviceMonitor labels to include release: prometheus label, which was originally at release: loki due to helm installation of loki being different from the helm installation of prometheus. This seems to have solved the problem. I can now see metrics in both prometheus and grafana (via prometheus datasource).