ViPR SRM | Watch4net: Additional Backend Is Listening but Filled with Stream Closed Errors

Various reports are missing metrics, properties. The Load Balancer and Arbiter are failing to push data to the Additional Backends, even though the backends show as started. Further investigation into the Additional Backend shows the following error in the logs:

WARNING -- [2017-01-10 15:16:15 EST] -- AbstractCollector::flushNext(): Error flushing Connector-Backend3
com.watch4net.apg.v2.collector.PipeError: Can't write to backend2.host.local:2300
   at com.watch4net.apg.v2.collector.plugins.SocketConnector.sendBuffer(SocketConnector.java:441)
   at com.watch4net.apg.v2.collector.plugins.SocketConnector.commit(SocketConnector.java:409)
   at com.watch4net.apg.v2.collector.plugins.SocketConnector.flushData(SocketConnector.java:452)
   at com.watch4net.apg.v2.collector.plugins.loadbalancer.decision.chain.AbstractEndPointElement.flushData(AbstractEndPointElement.java:63)
   at com.watch4net.apg.v2.collector.AbstractCollector.flushNext(AbstractCollector.java:85)
   at com.watch4net.apg.v2.collector.plugins.FailOverFilter.access$1400(FailOverFilter.java:53)
   at com.watch4net.apg.v2.collector.plugins.FailOverFilter$CheckNextComponentAvailabilityTimerTask.run(FailOverFilter.java:906) 
   at java.util.TimerThread.mainLoop(Timer.java:555)
   at java.util.TimerThread.run(Timer.java:505)
Caused by: java.net.ConnectException: Connection refused
   at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
   at com.watch4net.apg.v2.collector.plugins.SocketConnector.connect(SocketConnector.java:562)
   at com.watch4net.apg.v2.collector.plugins.SocketConnector.ensureConnected(SocketConnector.java:595)
   at com.watch4net.apg.v2.collector.plugins.SocketConnector.sendBuffer(SocketConnector.java:424)

The Additional Backend is currently not listening and is filled with the following errors:

WARNING -- [2017-01-03 10:31:28 EST] -- StorageGroup::handleRawData(): Unable to write raw value variable 'x.x.x.x.x.x.x.x.x'! java.io.IOException: Stream Closed

Restarting the Additional Backends host allowed the services to come up. However, in this particular case, the customer was unaware that the ABE had reached 100% file capacity. Some operations were no longer running as a result. The host must be cleaned up or more space must be added.

Troubleshooting steps included the following:

Run "netstat -anp | grep 2..." to verify that the backend sockets are listening on ports 2100, 2200, 2300, and 2400:

Host3:~ # netstat -anp | grep -i :2...
tcp 0 0 0.0.0.0:2100 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2101 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22   0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2200 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2201 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2300 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2301 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2400 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2401 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2050 0.0.0.0:* LISTEN