SUT:
- HAProxy (as balancer)
- WSO2 APIM (as API gateways)(one as a control-plane and 4 as gateways)
- microservice (written in Golang) in k8s
- CentOS-7 as a host OS for servers that runs HAProxy and WSO2 APIM.
For some reason in every test at some time (approx at 18 RPS (quite low load…)) appears a gap (RPS == 0)…
Then RPS start to grow again and periodically drops to some lower values, but not zero:
(this is example from another test, but such behavior appears constantly)
All logs (HAProxy, WSO2 APIMs, etc.) are empty, except WSO2 APIM gateways’ which showing such messages:
ERROR {org.apache.synapse.transport.passthru.SourceHandler} - I/O error: Connection reset java.net.SocketException: Connection reset
Metrics related to HTTP status-codes not showing any errors…
Consulting with sysadmins/SRE/security not revealed any hardware or software that may interfere…
k6 logs are ok too, except one strange thing:
at the time when gap appears:
my logging function from k6-script stops writing to console
there are just metrics’ related messages
and several following entries:
running (0h03m49.2s), 0335/3000 VUs, 1987 complete and 0 interrupted iterations
VUs keeps growing, but completed iterations stuck at 1987 for some reason, though there are 0 interrupted iterations…
Why I’m observing such behavior?
And what possible reasons could be for such behavior (RPS gap, no logs, connection reset errors, stucked at certain point with completed iterations) ?
