ViPR SRM: Alerting Module hangs

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms



Found that there were over 900 connections to port 2013 (some dynamic) that were in different states:

WARNING   [2017-04-10 03:49:42 EDT]   RawValueDecoder::decode(): Invalid raw value rejected
INFO   [2017-04-10 03:49:42 EDT]   ChannelNegotiationProtocol::transform(): [id: 0xbff120bb, //xx.xxx.xxx.xxx:46555 => //xx.xxx.xxx.xxx:2010] Initializing channel with no capabilities
INFO   [2017-04-10 03:49:49 EDT]   SocketSource::disconnect(): Dropping connection to /xx.xxx.xxx.xxx.
INFO   [2017-04-10 03:49:49 EDT]   SocketSource::connect(): Accepted incoming connection from /xx.xxx.xxx.xxx.
SRM version 4.0.1 vApp
Issues found during webex:
WARNING   [2017-04-10 03:49:42 EDT]   RawValueDecoder::decode(): Invalid raw value rejected
INFO   [2017-04-10 03:49:42 EDT]   ChannelNegotiationProtocol::transform(): [id: 0xbff120bb, //xx.xxx.xxx.xxx:46555 => //xx.xxx.xxx.xxx:2010] Initializing channel with no capabilities
INFO   [2017-04-10 03:49:49 EDT]   SocketSource::disconnect(): Dropping connection to /xx.xxx.xxx.xxx.
INFO   [2017-04-10 03:49:49 EDT]   SocketSource::connect(): Accepted incoming connection from /xx.xxx.xxx.xxx.
WARNING   [2017-04-10 03:49:49 EDT]   SocketSource$DataReaderWorker::run(): An incoming event could not be processed: com.watch4net.events.common.serialization.SerializationException: Malformed occurrence keys in event: HEAD / HTTP/1.0
WARNING   [2017-04-10 03:49:49 EDT]   SocketSource$DataReaderWorker::run(): An incoming event could not be processed: com.watch4net.events.common.serialization.SerializationException: Malformed definition keys in event: 
INFO   [2017-04-10 03:49:50 EDT]   BasicMessagesLoggingHandler::channelInactive(): [id: 0xbff120bb, /xx.xxx.xxx.xxx:46555 :> //xx.xxx.xxx.xxx:2010] Communication channel is now inactive/closed
INFO   [2017-04-10 03:49:50 EDT]   BasicMessagesLoggingHandler::channelActive(): [id: 0xf28e525d, //xx.xxx.xxx.xxx:47531 => //xx.xxx.xxx.xxx:2010] Communication channel is now active
SEVERE   [2017-04-10 03:49:50 EDT]   ApplicationDataForwarder::unhandledExceptionCaught(): An unhandled error occured on channel [id: 0xf28e525d, //xx.xxx.xxx.xxx:47531 => //xx.xxx.xxx.xxx:2010]
com.emc.watch4net.socket.communicator.handler.rawvalue.RawValueDecoder$InvalidRawValueExcepti
An incoming event could not be processed: com.watch4net.events.common.serialization.SerializationException: Malformed definition keys in event /xx.xxx.xxx.xxx pbe = Name:hostname.net Address: /xx.xxx.xxx.xxx
WARNING   [2017-04-12 05:14:26 EDT]   SocketSource$DataReaderWorker::run(): An incoming event could not be processed: com.watch4net.events.common.serialization.SerializationException: Malformed occurence keys in event: [1]ClientHello
WARNING   [2017-04-12 05:14:26 EDT]   SocketSource$DataReaderWorker::run(): An incoming event could not be processed: com.watch4net.events.common.serialization.SerializationException: Malformed occurence keys in event: EndMessage
INFO   [2017-04-12 05:14:34 EDT]   SocketSource::disconnect(): Dropping connection to 10.xxx.xxx.xxx.
INFO   [2017-04-12 05:14:34 EDT]   SocketSource::connect(): Accepted incoming connection from 10.xxx.xxx.xxx.
WARNING   [2017-04-12 05:15:46 EDT]   SocketSource$DataReaderWorker::run(): An incoming event could not be processed: com.watch4net.events.common.serialization.SerializationException: Malformed occurence keys in event: 
________________________________________

Cause

bunit-group.csv is invalid 

fe = xx.xxx.xxx.xxx Name: jxqpstgsrmfe01.onefiserv.net Address: xx.xxx.xxx.xxx
OPEN CONNECTIONS:
tcp 96848 0 xx.xxx.xxx.xxx:2013 xx.xxx.xxx.xxx:35817 ESTABLISHED off (0.00/0/0)
tcp 96848 0 1xx.xxx.xxx.xxx:2013 xx.xxx.xxx.xxx:55989 ESTABLISHED off (0.00/0/0)
tcp 96848 0 xx.xxx.xxx.xxx:2013 xx.xxx.xxx.xxx:60556 ESTABLISHED off (0.00/0/0)
tcp 30713 0 xx.xxx.xxx.xxx:2013 xx.xxx.xxx.xxx:50174 CLOSE_WAIT off (0.00/0/0)
tcp 43047 0 xx.xxx.xxx.xxx:2013 xx.xxx.xxx.xxx:38699 ESTABLISHED off (0.00/0/0)
tcp 22574 0 xx.xxx.xxx.xxx:2013 xx.xxx.xxx.xxx:58961 CLOSE_WAIT off (0.00/0/0)
tcp 98467 0 xx.xxx.xxx.xxx:2013 xx.xxx.xxx.xxx:43472 ESTABLISHED off (0.00/0/0)
tcp 0 50945 1xx.xxx.xxx.xxx:50364 xx.xxx.xxx.xxx:2013 FIN_WAIT1 unkn-4 (2.26/0/0)
Java Heap:
apg 32467 1 5 Apr04 ? 12:02:19 /opt/APG/Java/Sun-JRE/8.0.102/bin/java -Xms256m -Xmx2048m -javaagent:/opt/APG/bin/.runtime/service/1.10u4/apg-bootstrap-agent.jar -Djava.rmi.server.hostname=jxqpstgsrmbe01.onefiserv.net -Djava.util.logging.config.file=conf/alerting.logging.properties -Dcom.watch4net.utils.jmx.agent.config.file=conf/w4n-agent.properties -Dcom.watch4net.utils.jmx.agent.host=jxqpstgsrmbe01.onefiserv.net -javaagent:lib/w4n-jmx-agent.jar -cp /opt/APG/bin/.runtime/service/1.10u4/apg-service-bootstrap.jar:lib/* com.watch4net.apg.module.plugin.service.Bootstrap com.watch4net.alerting.engine.AlertingEngine main start

Resolution

The workaround provided there looks like this:
Add the workaround to apg.properties file under /opt/APG/bin of Primary backend, Additional backend and all the collector's if possible to know from which collector the data framelegnth is exceeding, although the pointer is at localhost but its good to add on every collector host. (Note: Make a backup of the apg.properties file first).
Add the following line to the bottom of the file: /opt/APG/bin/apg.properties of every SRM host (Primary BE, Additional BE, and collectors)
restricted.reader.line.size=50000
Then restart the Alerting backend services.

Affected Products

Storage Software
Article Properties
Article Number: 000050344
Article Type: Solution
Last Modified: 31 Jan 2025
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.