pool at capacity error

[jjg-123]

(See also https://jira.ncsa.illinois.edu/browse/CIL-1928)

This occurs because the pooling code is used for anything that has connections to manage, such as databases or http clients. The issue is that the counter for in use is not being correctly updated in certain failure modes. So the situation is that something starts failing (e.g. a service is down). The system them discards the connection, creates another and tries.

The problem is that depending on the failure type, the counter for connection in use was not being decremented, hence eventually there would be a "pool at capacity" error. What should happen is that the counter is decremented correctly and some logging about which component is failing should be done.

[jjg-123]

post mortem note: On issue is that the counter for closed connections was not being updated for HTTP pooling, leading to a false positive that it was running out of connections. And the messages are improved to refer to the component, so no generic error.