ViPR SRM: Waarschuwingsmodule loopt vast
Summary: Waarschuwingsmodule loopt vast in SRM
Symptoms
Gevonden dat er meer dan 900 verbindingen waren naar poort 2013 (sommige dynamisch) die verschillende statussen hadden:WARNING [2017-04-10 03:49:42 EDT] RawValueDecoder::decode(): Invalid raw value rejectedINFO [2017-04-10 03:49:42 EDT] ChannelNegotiationProtocol::transform(): [id: 0xbff120bb, //xx.xxx.xxx.xxx:46555 => //xx.xxx.xxx.xxx:2010] Initializing channel with no capabilitiesINFO [2017-04-10 03:49:49 EDT] SocketSource::disconnect(): Dropping connection to /xx.xxx.xxx.xxx.INFO [2017-04-10 03:49:49 EDT] SocketSource::connect(): Accepted incoming connection from /xx.xxx.xxx.xxx.SRM version 4.0.1 vAppIssues found during webex:WARNING [2017-04-10 03:49:42 EDT] RawValueDecoder::decode(): Invalid raw value rejectedINFO [2017-04-10 03:49:42 EDT] ChannelNegotiationProtocol::transform(): [id: 0xbff120bb, //xx.xxx.xxx.xxx:46555 => //xx.xxx.xxx.xxx:2010] Initializing channel with no capabilitiesINFO [2017-04-10 03:49:49 EDT] SocketSource::disconnect(): Dropping connection to /xx.xxx.xxx.xxx.INFO [2017-04-10 03:49:49 EDT] SocketSource::connect(): Accepted incoming connection from /xx.xxx.xxx.xxx.WARNING [2017-04-10 03:49:49 EDT] SocketSource$DataReaderWorker::run(): An incoming event could not be processed: com.watch4net.events.common.serialization.SerializationException: Malformed occurrence keys in event: HEAD / HTTP/1.0WARNING [2017-04-10 03:49:49 EDT] SocketSource$DataReaderWorker::run(): An incoming event could not be processed: com.watch4net.events.common.serialization.SerializationException: Malformed definition keys in event: INFO [2017-04-10 03:49:50 EDT] BasicMessagesLoggingHandler::channelInactive(): [id: 0xbff120bb, /xx.xxx.xxx.xxx:46555 :> //xx.xxx.xxx.xxx:2010] Communication channel is now inactive/closedINFO [2017-04-10 03:49:50 EDT] BasicMessagesLoggingHandler::channelActive(): [id: 0xf28e525d, //xx.xxx.xxx.xxx:47531 => //xx.xxx.xxx.xxx:2010] Communication channel is now activeSEVERE [2017-04-10 03:49:50 EDT] ApplicationDataForwarder::unhandledExceptionCaught(): An unhandled error occured on channel [id: 0xf28e525d, //xx.xxx.xxx.xxx:47531 => //xx.xxx.xxx.xxx:2010]com.emc.watch4net.socket.communicator.handler.rawvalue.RawValueDecoder$InvalidRawValueExceptiAn incoming event could not be processed: com.watch4net.events.common.serialization.SerializationException: Malformed definition keys in event /xx.xxx.xxx.xxx pbe = Name:hostname.net Address: /xx.xxx.xxx.xxxWARNING [2017-04-12 05:14:26 EDT] SocketSource$DataReaderWorker::run(): An incoming event could not be processed: com.watch4net.events.common.serialization.SerializationException: Malformed occurence keys in event: [1]ClientHelloWARNING [2017-04-12 05:14:26 EDT] SocketSource$DataReaderWorker::run(): An incoming event could not be processed: com.watch4net.events.common.serialization.SerializationException: Malformed occurence keys in event: EndMessageINFO [2017-04-12 05:14:34 EDT] SocketSource::disconnect(): Dropping connection to 10.xxx.xxx.xxx.INFO [2017-04-12 05:14:34 EDT] SocketSource::connect(): Accepted incoming connection from 10.xxx.xxx.xxx.WARNING [2017-04-12 05:15:46 EDT] SocketSource$DataReaderWorker::run(): An incoming event could not be processed: com.watch4net.events.common.serialization.SerializationException: Malformed occurence keys in event: ________________________________________
Cause
bunit-group.csv is ongeldig
fe = xx.xxx.xxx.xxx Name: jxqpstgsrmfe01.onefiserv.net Address: xx.xxx.xxx.xxxOPEN CONNECTIONS:tcp 96848 0 xx.xxx.xxx.xxx:2013 xx.xxx.xxx.xxx:35817 ESTABLISHED off (0.00/0/0)tcp 96848 0 1xx.xxx.xxx.xxx:2013 xx.xxx.xxx.xxx:55989 ESTABLISHED off (0.00/0/0)tcp 96848 0 xx.xxx.xxx.xxx:2013 xx.xxx.xxx.xxx:60556 ESTABLISHED off (0.00/0/0)tcp 30713 0 xx.xxx.xxx.xxx:2013 xx.xxx.xxx.xxx:50174 CLOSE_WAIT off (0.00/0/0)tcp 43047 0 xx.xxx.xxx.xxx:2013 xx.xxx.xxx.xxx:38699 ESTABLISHED off (0.00/0/0)tcp 22574 0 xx.xxx.xxx.xxx:2013 xx.xxx.xxx.xxx:58961 CLOSE_WAIT off (0.00/0/0)tcp 98467 0 xx.xxx.xxx.xxx:2013 xx.xxx.xxx.xxx:43472 ESTABLISHED off (0.00/0/0)tcp 0 50945 1xx.xxx.xxx.xxx:50364 xx.xxx.xxx.xxx:2013 FIN_WAIT1 unkn-4 (2.26/0/0)Java Heap:apg 32467 1 5 Apr04 ? 12:02:19 /opt/APG/Java/Sun-JRE/8.0.102/bin/java -Xms256m -Xmx2048m -javaagent:/opt/APG/bin/.runtime/service/1.10u4/apg-bootstrap-agent.jar -Djava.rmi.server.hostname=jxqpstgsrmbe01.onefiserv.net -Djava.util.logging.config.file=conf/alerting.logging.properties -Dcom.watch4net.utils.jmx.agent.config.file=conf/w4n-agent.properties -Dcom.watch4net.utils.jmx.agent.host=jxqpstgsrmbe01.onefiserv.net -javaagent:lib/w4n-jmx-agent.jar -cp /opt/APG/bin/.runtime/service/1.10u4/apg-service-bootstrap.jar:lib/* com.watch4net.apg.module.plugin.service.Bootstrap com.watch4net.alerting.engine.AlertingEngine main start
Resolution
De tijdelijke oplossing ziet er als volgt uit:
Voeg de tijdelijke oplossing toe aan apg.properties file onder /opt/APG/bin van Primaire backend, Aanvullende backend en alle collector's indien mogelijk om te weten van welke collector de data afkomstig is framelegnth is overschrijding, hoewel de pointer is op localhost, maar het is goed om toe te voegen op elke collector host. (Opmerking: Maak eerst een back-up van het apg.properties bestand).
Voeg de volgende regel toe aan de onderkant van het bestand: /opt/APG/bin/apg.properties van elke SRM-host (Primary BE, Additional BE en collectors)restricted.reader.line.size=50000
Start vervolgens de Alerting backend-services opnieuw.