Caddy v2.8.4/v2.10.0 Randomly Enters 502 State Across All Domains, Possibly SSE-Related

[syedrizwansrs]

We are experiencing a problem where Caddy replies with 502's, the server appears to transition into this locked state where any request sent to any hosted domain even on separate IP addresses with separate back-end's will return a 502, dispite the backend servers being online.

This problem does not directly correlate with high load, or with duration that the server is running. However one correlating factor was a recent introduction of SSE, which maybe 50 max SSE connections would be active on one of the domains hosted.

We are using version v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk= and also tried version v2.10.0 h1:fonubSaQKF1YANl8TXqGcn4IbIRUDdfAkpcsfI/vX5U=, both exhibit the same problem.

The config we have is as follows (IP's and domains obfuscated):

caddy-sample-config.txt

We have currently written a script which checks the logs for a large number of 502's and restarts the server, but we can catch this scenario and take the BPG route off for this server. So if there is any specific debugging that you would like us to run when it happens, please let us know.

Thanks