How to troubleshoot SNMP target not receiving SNMP alerts sent from Unisphere.

Below are the four places to check if an SNMP target fails to receive SNMP alerts sent from Unisphere.
 

• The community word of customer’s SNMP target should match the default community word (SNMP_trap) of Unisphere.  
• Customer must ensure good network connectivity between Unisphere server and SNMP target. We can use openssl and tracepath commands on Unisphere server to verify the connectivity and use tcpdump command on Unisphere server to capture the SNMP package during alert tests. A straightforward way to prove Unisphere can send out SNMP alerts successfully is to have an SNMP target directly connected to the Unisphere server. If the directly connected SNMP target can receive SNMP alerts while the nondirectly connected cannot, it means that the issue is with customer’s network.  
• Unisphere SNMP alert settings should be all set correctly, which include:    
• SNMP notification config, Alert policy setting, Notification level.

 
If the above have no issues, support must proceed with alerts test with SE storevntd debug log enabled, smas INFO logging set, and tcpdump captured. Collect emcgrab and test results if we cannot see evidence that SNMP alerts have been successfully sent out.
Note, it is better to generate real alerts other than using the "TEST SNMP" function during the test, because compared with real alerts, "TEST SNMP" package is small and more unreliable to transmit for UDP package.

 

Detailed steps for troubleshooting: 

1. SNMP Community word.

The default community word is "SNMP_trap". If it does not match the community word of the SNMP target, we can change it by adding the following line in SE daemon_options file:
 

storevntd: snmp_trap_community=<community word of SNMP target>        
(example: storevntd: snmp_trap_community=public)

Save the file, and then restart the storevntd daemon to let the change take effect. (Do not use the reload command since it is not reliable)

stordaemon shutdown storevntd
stordaemon start storevntd

Once storenvtd is restarted, the alert policies previously loaded from Unisphere alert configuration page also get lost. In order to let the alert policies get reloaded and registered in storevntd, we must restart smas service as well.
(The command below is for restarting smas service in Linux server. You can follow SNMP events are not being sent to Embedded U4V or SNMP server to restart smas service for embedded Unisphere).

On Unisphere server, cd to the path /etc/init.d, and then run the following commands:

./smas stop
./smas start

Then we can use command below to check the storevntd event registration status. (It takes a bit time for events registered into storevntd after smas restart)

stordaemon action storevntd -cmd list -regs –v

 

2. Network connectivity.

We can use openssl and tracepath command on Unisphere server to check if Unisphere can communicate with SNMP target well.
 

openssl s_client  -connect <SNMP target IP>:<port>

tracepath <SNMP target IP>:<port> 
(in some version of tracepath, the format is “tracepath <SNMP target IP>/<port>” )

Examples:

openssl s_client  -connect xx.xxx.0.xx:162
Connected

tracepath xx.xxx.0.xx -p 162

1?: [LOCALHOST]                                        pmtu 1500
1:  xxx.1x.xxx.xx0                                        0.854ms
1:  xxx.x8.xxx.xx0                                        0.540ms
2:  xx.xx1.xx.x                                             18.572ms
3:  xx.xxx.0.xx                                             0.988ms reached

 

3. Unisphere SNMP alert settings:

 

1. SNMP alert notification config.  

Settings - Alerts - Notifications, click on the "Configure" word on the SNMP block and ensure that the correct SNMP target IP and port are configured.

(Note: Test with a single SNMP target to avoid test complexity.)



To verify if these configurations are successfully loaded into storevntd daemon, we can use the following command to check:

stordaemon getvar storevntd -name smas_log_event_targets

storevntd                     smas_log_event_targets=snmp

stordaemon getvar storevntd -name snmp_host

storevntd                     snmp_host=O:xx.xx.xxx.xx:162:v1

stordaemon action storevntd -cmd list -log_targets

(for syslog alert, the first two corresponding commands are:

stordaemon getvar storevntd -name log_event_syslog_host
stordaemon getvar storevntd -name log_event_syslog_port

 

2. Alert policy setting

Later we are going to generate a real device config change alert for the alert test, so here, we use "device config change" policy as an example. Ensure "Device Config Change" is checked for the array used for the alert test and notifications type is specified with SNMP.



 

3. Notification level

By creating a new device for the alert test, the alert we generate is "Device configuration has changed". The alert is at the information level, so we must ensure that the blue icon is selected for an alert test at least.


 

4. Enable storevntd debug log and Unisphere INFO log

 Change the following parameters in SE daemon_options file:

storevntd:LOG_LEVEL = debug
storevntd:LOGFILE_TYPE = dated
storevntd:LOGFILE_RETENTION = 7

Save the file, as per step 1; we must restart storevntd daemon and then restart the smas service.
After smas service is restarted, change the logging level of Unisphere to INFO. (Note: Restarting smas service changes the logging level back to default, which is WARN. )
 

5. Alert test

Generate a real alert, use tcpdump to capture SNMP package and check storevntd debug log. 

1. Run the tcpdump command on Unisphere server to get ready for SNMP package capturing:

tcpdump -i any udp port 162 

2. Run the following command to check the existing SNMP event counter:

stordaemon action storevntd -cmd list -log_stats

storevntd                     Event Loggers
file             : 0 events delivered
system           : 0 events delivered
syslog           : 17 events delivered
snmp             : 4 events delivered

3. Create a 100M LUN in Unisphere. Check the Unisphere alert panel, and you see that "Device configuration has changed" alert is generated. If you cannot see the alert pop out, click the refresh button on the top bar of Unisphere’s page.

 (If you also enabled "device status" policy, you can also see "Device state has changed to Online" alert is generated).


 

4. Check the output of tcpdump.

The following shows that the SNMP alert is successfully captured:
 

tcpdump -i any udp port 162
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
11:05:59.246606 IP MGMT-0.emc-gateway > xx.xxx.xx.xxx.snmptrap:  C="SNMP_trap" Trap(448)  E:1139.3 xx.xxx.xx.xxx enterpriseSpecific s=4 0 X:xx.1.xx.1.8.xx.6.4.xxx.xxx.8x.xxx.6x.0.0.0.0.0.0.0.0.7=7 X:xx.1.11.1.7.xx.6.4.xxx.xxx.xx.xxx.xx.0.0.0.0.0.0.0.0.7=2 X:xx.1.xx.1.8.xx.6.4.xxx.xxx.8x.xxx.6x.0.0.0.0.0.0.0.0.7=.0.0 X:xx.1.11.1.9.xx.6.4.xxx.xxx.xx.xxx.xx.0.0.0.0.0.0.0.0.7="Symmetrix 000xxxx0xxxx : Device configuration has changed. - Object is: 000xxxx0xxxx:00867" X:xx.1.xx.1.6.xx.6.4.xxx.xxx.xx.xxx.xx.0.0.0.0.0.0.0.0.7=8 X:xx.1.6.1.20.xx.6.4.xxx.xxx.xx.xxx.xx.0.0.0.0.0.0.0.0="000xxxx0xxxx" X:xx.1.6.1.3.xx.6.4.xxx.xxx.xx.xxx.xx.0.0.0.0.0.0.0.0=11 E:1139.3.8888.1.0=2 E:1139.3.8888.2.0=5000 E:1139.3.8888.3.0=0 E:1139.3.8888.4.0="" 

5. Check the event counter again. We can see one more snmp event delivered.

stordaemon action storevntd -cmd list -log_stats

storevntd                     Event Loggers
file             : 0 events delivered
system           : 0 events delivered
syslog           : 17 events delivered
snmp             : 5 events delivered

6. Check the storenvtd debug log:

If you can find the following entries in the log, it shows storevntd has successfully delivered the SNMP alert:

<Dbg  > [4964             EvtLogger] Feb-15 11:05:59.246 : [sendTrapForEach()] Trap sent to xx.xxx.xx.xxx, port 162

<Dbg  > [4964             EvtLogger] Feb-15 11:05:59.246 : [processOneSymmEvent()] Symmetrix 000xxxxxxx1 : Device configuration has changed. - Object is: 000xxxxxxxx1:00024
<Dbg  > [4964             EvtLogger] Feb-15 11:05:59.246 : [processAsyncEvent()] Function Exit, rc= 470
<Dbg  > [4964             EvtLogger] Feb-15 11:05:59.246 : [evtd_logLoggerThrd] Event Delivery complete

The evidence above shows that SNMP alert has been successfully sent out from Unisphere side. If we fail to capture the SNMP package using tcpdump and storevntd also does not have evidence for successful event delivery, collect the emcgrab, and engage Technical support.