Reverse proxy configuration via haproxy

Grafana
1

I want to put few services behind reverse proxy, simple services work.
The 502 issue occurs when trying to reach grafana simple json data source (POST + json payload).
Grafana itself is not behind reverse proxy.

current haproxy config:

frontend FRONT.AWS.WEB.PROXY
mode http
bind *:8080
timeout client 1m
option  httplog
acl IS_RELE path_beg /release
acl IS_GRAF path_beg /grafana
use_backend BACK.AWS.WEB.ARTIFACTS if IS_RELE
use_backend BACK.AWS.WEB.RTGRAF if IS_GRAF

backend BACK.AWS.WEB.ARTIFACTS
mode http
http-request set-path /
http-response replace-value X-Application-Context (.*)(\release).*$ \1
server AWS.WEB.ARTIFACTS *:5581/ maxconn 1000 check port 5581

backend BACK.AWS.WEB.RTGRAF
mode http
#option forwardfor
#balance source
#option httpclose
#option httpchk HEAD / HTTP/1.0
http-request set-path /
http-response replace-value X-Application-Context (.*)(\grafana).*$ \1  
server AWS.WEB.RTGRAF *:5582/ maxconn 1000 check port 5582

data source config in grafana:

http://192.168.56.101:8080/grafana/

This is working request without haproxy:

curl -d '{"requestId":"Q119","timezone":"utc".....lters":[]}' -H 'Content-Type: application/json' http://localhost:8080/query

good response:

[{"columns":[{"text":"sym","type":"string"}, {"text":"time","type":""}, .... .... {"text":"mode","type":"string"}, {"text":"proto","type":"string"}],"rows":[],"type":"table"}]

BUT with haproxy:

 curl -d .... http://localhost:8080/grafana/query

502 Response:

<h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.

but just to confirm, service itself works:

curl http://localhost:8080/grafana/?2+1

Response:

<html><head><style>a{text-decoration:none}a:link{color:024C7E}a:visited{color:024C7E}a:active{color:958600}body{font:10pt verdana;text-align:justify}</style></head><body><pre>3

Haproxy log:

127.0.0.1:42362 [16/Sep/2020:21:57:14.430] FRONT.AWS.WEB.PROXY BACK.AWS.WEB.RTGRAF/AWS.WEB.RTGRAF 0/0/0/3/3 200 274 - - ---- 1/1/0/0/0 0/0 "GET /grafana/?2+1 HTTP/1.1"                            
127.0.0.1:42418 [16/Sep/2020:21:57:32.038] FRONT.AWS.WEB.PROXY BACK.AWS.WEB.RTGRAF/AWS.WEB.RTGRAF 0/0/0/-1/0 502 214 - - PH-- 1/1/0/0/0 0/0 "POST /grafana/query HTTP/1.1"

grafana log:

INFO[09-16|22:23:02] Request Completed logger=context userId=1 orgId=1 uname=admin method=POST path=/api/datasources/proxy/2/query status=502 remote_addr=192.168.56.1 time_ms=6 size=107 referer="http://192.168.56.101:3000/d/aQPWEJFmz/system-status?orgId=1&refresh=10s"