File provisioned alert not deleting

I have an alert that was created through a k8s configmap. I have since deleted the configmap, which as expected also deleted the alert file from the grafana container. Despite that, the alert is not being deleted from grafana itself and is still firing. The ui and api don’t let me delete it, since it was provisioned by a file, and the file is already gone so I can’t remove it that way. Are there any other options or is this a bug?