Loki mixin: LokiRequestLatency alert got fired

Hi everyone, recently we just enabled the Loki mixin from upstream: loki/production/loki-mixin at main · grafana/loki · GitHub, and LokiRequestLatency (loki/alerts.libsonnet at main · grafana/loki · GitHub) gets fired constantly since the p99 latency of querier and query frontend for my environment sometimes is around 4s.
My questions are mainly:

  1. How to solve this permanently? does scale up/out querier/query-frontend help?
  2. The severity of this alert is critical, does it deserve it? or warning should be OK?