This white paper provides configuration considerations and best practices of Dell PowerScale OneFS Alerting including the following topics:

• The detailed configuration considerations covering all three types of alerts in OneFS:

• SMTP (Simple Mail Transfer Protocol) alerts
• SNMP (Simple Network Management Protocol) TRAP

• General alert configuration considerations and typical alert configuration scenarios
• Detailed configuration considerations for the OneFS SNMP monitoring feature
• Useful tools and CLI commands introduction and explanation
• The complete SNMP TRAP list for OneFS
• The complete query list for the SNMP monitoring feature

This guide is intended for experienced system and storage administrators who are familiar with file services and network storage administration. This guide assumes the reader has a working knowledge of the following:

• Network-attached storage (NAS) systems
• The SNMP and SMTP protocols
• The PowerScale scale-out storage architecture and the PowerScale OneFS operating system

Readers should also be familiar with PowerScale documentation resources including:

• Dell Community Network info hubs
• Dell OneFS release notes, which are available on the Dell support network and contain important information about resolved and known issues.
• Dell PowerScale OneFS Best Practices

Dell Technologies and the authors of this document welcome your feedback on this document. Contact the Dell Technologies team by email.

Author: Vincent Shen

Note: For links to other documentation for this topic, see the PowerScale Info Hub.

This section focuses on the types of alert notifications that OneFS supports:

• SMTP alerts
• SNMP TRAP
• ConnectEMC alerts

For each type of alert, the specific architecture and detailed configuration considerations will be discussed and explained in the following sections. Apart from that, this section will also introduce some general alert configuration considerations which apply to all the three types of alerts. Finally, it will provide some typical examples to configure alerts in OneFS.

Figure 1 shows the overall architecture of the alert system in OneFS.

Figure 1.   Alert architecture

Event Capture is the first stage in the processing pipeline. It is responsible for reading event occurrences from the kernel queue, storing them safely on persistent local storage, generating attachments and queueing them in priority buckets for analysis.

Event Analysis is the second stage in the processing pipeline. The main analysis process collects related event occurrences together as event group occurrences, which can be processed by the next stage – Reporter.

Reporter is the third stage in the processing pipeline. The reporter periodically queries Event Analysis for event group occurrences which have changed. For each of these changes, Reporter evaluates any relevant alert conditions, generating alert requests for any events which are satisfied to the next stage – Alerter.

Alert Condition defines how event group occurrences are alerted. An alert condition states that a specified set of event groups will be reported on by a specified set of channels under a specified condition as described in Table 1:

Table 1.      Alert condition types

Alerter is the final stage in the processing pipeline. It is responsible for actually delivering the alerts requested by the reporter.

Channel is a named destination for alerts. A channel specifies the mechanisms by which alerts are sent, which are listed below:

• SMTP
• SNMP TRAP
• ConnectEMC

Figure 2 shows the relationship between event group categories, event types, and event groups.

Figure 2.   Event group category, event type, and event group

Event group category defines 10 subsets of events at a very high level as listed in Table 2. For each event group category, there are several events defined. An event definition is also known as an event type which is for a specific type of event. Event groups contain closely related event occurrences.

Table 2.      Event group categories

Event groups are collections of individual events that shares related symptoms of a single situation on the PowerScale cluster. Event groups provide a single point of management for multiple event instances that are generated in response to a specific situation on the cluster. Starting from OneFS 8.0, the related events are organized into event groups. In this case, the CLI command isi event is an abbreviation for isi event group.

The following sections list some general alert configuration considerations including the following:

• Alert granularity customization
• Alert severity configuration
• Alert configuration by the deny list
• Stop receiving alert notifications for heart beat events
• Cause Long vs. Cause Short

Alert granularity customization

As covered in the section Alert architecture, events are organized into event groups and event groups are categorized by the event group categories. Alerts can be configured in different granularities by event group categories or event groups through the web UI and the CLI command. The following examples demonstrate alert configuration using both of these granularities.

The following command creates an alert named demo_alerts, sets the alert condition to NEW and sets the event group categories to System disk events and Node status events subscribing to all new event groups in these categories. To view all the event group categories, see Table 2.

isi event channels create mychannel smtp --address my_email@xxx.com --smtp_host smtp.xxx.com

isi event alerts create demo_alerts --category "100000000,200000000" NEW mychannel

The following command creates an alert named demo_alerts, sets the alert condition to NEW and only subscribes to event groups 100010001 and 100010009.

isi event alerts create demo_alerts --eventgroup "100010001,100010009" NEW mychannel

To view all the event groups which can be subscribed through SMTP in the form of the combination of event group ID, event group name, event group description and the belonging event group category, use the following CLI command:

isi event types list –format list

Sample output:

--------------------------------------------------------------------------

ID: 900100017

Name: HW_NVRAM_SRAM_ECC_CORRECTABLE

Category: 900000000

Description: NVRAM SRAM correctable (single-bit) ECC error in chassis {chassis} slot {slot}

--------------------------------------------------------------------------

Note: The command isi event types list is only available in OneFS 8.0.0.5 and later versions.

Alert severity configuration

It is highly recommended to tune the alert severity based on each specific environment to minimize unnecessary email alerts. The only way to configure alert severity is to use the OneFS CLI.

The following command creates an alert named ExternalNetwork, sets the alert condition to NEW, sets the source event group to the event group with the ID number 400160001, sets the channel that will broadcast the event to RemoteSupport and sets the severity level to critical:

isi event alerts create ExternalNetwork NEW --eventgroup 400160001 --channel RemoteSupport --severity critical

The following example modifies the alert named ExternalNetwork to ExtNetwork, adding the event group with an event group ID number of 400160001, and filtering so that alerts will only be sent for event groups with a severity value of critical:

isi event alerts modify ExternalNetwork --name ExtNetwork --add-eventgroup 400160001 --severity critical

Alert configuration by the deny list

It is a quite common scenario to configure alerts by deny list which means to alert on all the types of event groups except type X, Y, and Z. The following examples will demonstrate how to achieve this function by a custom script. This example will subscribe to all event group types except 400160001 in the SMTP alerts configuration. The implementation is different between OneFS versions before 8.0.0.5 and later:

1. In OneFS version 8.0.0.0 – 8.0.0.4, create a file called eventgroups.py containing the following code. This is to generate a collection of all the event group types.

#!/usr/bin/python

import json

efile="/etc/celog/events.json"

egfile="/etc/celog/eventgroups.json"

with open(efile, "r") as ef:

    e = json.loads(ef.read())

with open(egfile, "r") as egf:

    eg = json.loads(egf.read())

out = []

symptoms = []

for eg,v in eg.iteritems():

    if eg != v["name"]:

        print "warning.. name not same %s" % eg

    out += [eg]

    for s in v["symptoms"]:

        symptoms += [s]

for e,v in e.iteritems():

    if e not in symptoms:

         out += [e]

print ",".join(out)

2. Create an SMTP channel and associate it with a customized alert to exclude event group type 400160001.

isi event channels create mychannel smtp --address my_email@xxx.com --smtp_host smtp.xxx.com

isi event alert create myalert NEW mychannel --eventgroup `./eventgroups.py | sed 's/,400160001//'`

3. To verify the configuration, send a test alert by leveraging the following script. The expected outcome is that the SMTP alert for event group type 40005002 is sent to the subscriber.

/usr/bin/isi_celog/celog_send_events.py -o 400160001

/usr/bin/isi_celog/celog_send_events.py -o 400050002

In 8.0.0.5 and later versions, the list of event group types is available by isi event type list. It will do the same job as the scripts in the above step 1 to generate a collection of all the event group types. Setting up an SMTP alert for all event group types except 40016001 is described below:

1. Create an SMTP channel and associate it to a customized alert to exclude event group type 400160001.

isi event channels create mychannel smtp --address my_email@xxx.com --smtp_host smtp.xxx.com

isi event type list --format json | python -c 'import json;import sys;print ",".join([eg["id"] for eg in json.loads(sys.stdin.read())])' | sed 's/,400160001//' > a.txt

isi event alert create myalert NEW mychannel --eventgroup `cat a.txt`

2. To verify the configuration, send the test alert by leveraging the following script. The expected outcome is that the SMTP alert for event group type 40005002 is sent to the subscriber.

/usr/bin/isi_celog/celog_send_events.py -o 400160001

/usr/bin/isi_celog/celog_send_events.py -o 400050002

Starting in OneFS 9.4.0.0, you can use PAPI to achieve the same purpose. Send event IDs that you want to exclude in an HTTP PUT request to /platform/15/event/alert-conditions/<alert rule name>

The following is an example data body:

{"exclude_eventgroup_ids": ["100010001", "100010002"]}

The following figure shows an example to exclude the event ID 100010001 and 100010002 from the alert – DemoAlertRule:

Figure 3.   An example to exclude the events from an alert

Stop receiving alert notification for heartbeat events

In OneFS 8.0 and later versions, PowerScale cluster will generate a number of heartbeat events every day. Heartbeat alerts are intended to tell you when a node is unable to send alerts. Normally these events are only for informational purposes. The total number of heartbeat alert notifications is equal to the total number of nodes in the PowerScale cluster. For example, if you have configured SMTP alerts in a 60-node PowerScale cluster; you will get 60 email alert notifications for heart beat events every day. The event group ID for heart beat events is 400050004. So you can either add event group ID 400050004 to the deny list or you can set a non-informational severity for this event as discussed in the section Alert severity configuration. The following examples will demonstrate both ways.

Use the following CLI commands to exclude the heart beat event group (400050004) in the SMTP alert configuration. (OneFS 8.0.0.5 and later)

isi event channels create mychannel smtp --address my_email@xxx.com --smtp_host smtp.xxx.com

isi event type list --format json | python -c 'import json;import sys;print ",".join([eg["id"] for eg in json.loads(sys.stdin.read())])' | sed 's/,400050004//' > a.txt

isi event alert create myalert NEW mychannel --eventgroup `cat a.txt`

Use the following CLI command to set the severity level for heart beat event group (400050004) to emergency, critical, and warning.

isi event alerts create demo_alerts --eventgroup "400050004" --severity "emergency,critical,warning" NEW mychannel

Causes long or causes short

Starting from OneFS 8.0, the related events are organized into event groups. In this case, the CLI command isi event is an abbreviation for isi event group. To view a list of groups of correlated event occurrences, we can use either isi event list or isi event groups list to achieve this. This command will list a short description (causes short) for each event group as shown in Figure 4.

Figure 4.   Causes short

If more details are preferred, we recommend you use the CLI command isi event list -v or isi event group list -v to view the long description (Causes Long). Here is another example:

Figure 5.   Causes long

Configurable event threshold

This feature is introduced in OneFS 9.1.0.0 and allows users to customize event thresholds to a level listed below other than the defaults.

• Info
• Warn
• Crit.
• Emerg.

In OneFS 9.1.0.0, seven events are configurable with their thresholds. For the configurable events and their default threshold values, see Table 3:

Table 3.      Configurable event thresholds and their default values

To configure the event thresholds, use WebUI or CLI as the following:

Figure 6.   WebUI to configure the event thresholds

The following are examples of how to use CLI to list, view, modify, and reset the event thresholds:

vshen-llh7eh7-1# isi event thresholds view 100010001

         ID: 100010001

    ID Name: SYS_DISK_VARFULL

Description: Percentage at which /var partition is near capacity

   Defaults: info (75%), warn (85%), crit (90%)

 Thresholds: info (75%), warn (85%), crit (90%)

vshen-llh7eh7-1# isi event thresholds modify 100010001 --info 50 --crit 95

vshen-llh7eh7-1# isi event thresholds view 100010001

         ID: 100010001

    ID Name: SYS_DISK_VARFULL

Description: Percentage at which /var partition is near capacity

   Defaults: info (75%), warn (85%), crit (90%)

 Thresholds: info (50%), warn (85%), crit (95%)

vshen-llh7eh7-1# isi event thresholds reset 100010001

Are you sure you want to reset info, warn, crit from event 100010001?? (yes/[no]): yes

vshen-llh7eh7-1# isi event thresholds view 100010001

         ID: 100010001

    ID Name: SYS_DISK_VARFULL

Description: Percentage at which /var partition is near capacity

   Defaults: info (75%), warn (85%), crit (90%)

 Thresholds: info (75%), warn (85%), crit (90%)

New WebUI for CELOG

In OneFS 9.2.0.0, the user interface (UI) for CELOG has been re-designed to make sure everything is in good order and easy to use. You can easily:

• Show events for

• Today
• This week
• This month
• Custom range/
• All

• Categorize all the events by their status like

• Active
• Ignored
• Resolved
• All

• Filter the event by the severity like

• Emergency
• Critical
• Warning
• Information

• Search for a specific event in the event history
• Resolve a bulk of events
• Ignore a bulk of events

Figure 7 shows a screenshot of the new WebUI for Event group history. You can get to this new UI by clicking Event and alerts under Cluster management in OneFS WebUI. Please click around and let us know if you have any comments or suggestions about it.

Figure 7.   New WebUI for Event group history

CELOG maintenance mode

In OneFS 9.2.0.0, you can manually enable and disable CELOG maintenance mode. During a CELOG maintenance window, the system will continue to log events, but no alerts will be generated. You will have the opportunity to review all events that took place during the maintenance window when disabling maintenance mode. Active event groups will automatically resume generating alerts when the scheduled maintenance period ends.

The following steps will lead you to go through this feature:

1. To enable CELOG maintenance mode, in OneFS WebUI, click Event and alerts under Cluster management. Click the Alert management tab, and click Enable CELOG maintenance mode. In the prompt window, click Enable CELOG maintenance mode shown in Figure 8:

Figure 8.   Enable CELOG maintenance mode

2. Create an Alert channel for testing. This operation depends on your specific environment, for example, you can either choose SMTP or SNMP channel as what you have pre-configured in your environment. In this example, I create an SMTP channel in the lab. To create a channel, click Create channel under Alert channel in the CELOG alerting section as shown in Figure 9.

Figure 9.   Create channel

3. To create an alert rule, click the tab Alerting rule and then click the button Create alert rule
4. In the prompt window, fill the Rule name, set the Rule condition to NEW, apply it to all the Alert categories, and attached it to the channel you have just created.

Figure 10.    Rule configuration

5. Open an SSH connection to any node in the cluster and log in using the root account. Use the following command to create several events:

# /usr/bin/isi_celog/celog_send_events.py -o 940100002

The output is like below:

Heap looptimes: [-1]

running -1 [940100002]

1612871342 :: Sending eventids [940100002] with specifier None

940100002 message is OneFS {version} is currently running on unsupported nodes (devid(s) {devids}). {msg}.

1.195 (70368744177859) corresponds to eventid 940100002

Out of events to run. Exiting.

# /usr/bin/isi_celog/celog_send_events.py -o 940100001

The output is like below:

Heap looptimes: [-1]

running -1 [940100001]

1612872343 :: Sending eventids [940100001] with specifier None

940100001 message is OneFS {version} is currently running and is not supported on this hardware: {msg}.

1.196 (70368744177860) corresponds to eventid 940100001

Out of events to run. Exiting.

6. During maintenance mode, OneFS will still show the event but there will be no alert. In this example, there is no SMTP alert email triggered. In the Event group history, this event is shown in Figure 11.

Figure 11.    Event in history

You can also use the following command to filter all the events which happened during the CELOG maintenance mode:

# isi event groups list --maintenance-mode=true

The output is like the following:

ID   Started     Ended       Causes Short                           Lnn  Events  Severity

------------------------------------------------------------------------------------------

16   02/09 11:49 --          HW_CLUSTER_ONEFS_VERSION_NOT_SUPPORTED 1    1       critical

17   02/09 12:05 02/09 12:19 HW_ONEFS_VERSION_NOT_SUPPORTED         1    1       critical

------------------------------------------------------------------------------------------

7. Click the button Disable CELOG maintenance mode. In the following prompt window, it will list all the events during the CELOG maintenance mode and you can:
   1. View event details
   2. Ignore event
   3. Resolve event

Figure 12.    Disable CELOG maintenance mode

In this example, the event HW_ONEFS_VERSION_NOT_SUPPORTED is marked resolved by clicking Action and Resolve event.

After the CELOG maintenance mode is disabled, you will get the email notification only for HW_CLUSTER_ONEFS_VERSION_NOT_SUPPORTED. The event which has been marked resolved will not trigger any notification.

Suppress Event Notification

When an event type is suppressed, it prevents an event from alerting on all configured CELOG channels. But the event will still show in the event group history.

The following steps will lead you to go through this feature:

1. To suppress an event type, click the button Suppress for a specific event under the Event type ID tab as shown in Figure 13. In this example, both 930100006 and 930100005 have been suppressed.

Figure 13.    Suppress events

2. Open an SSH connection to any node in the cluster and log in using the root account. Use the following command to create several events:

# /usr/bin/isi_celog/celog_send_events.py -o 930100006

The output is:

Heap looptimes: [-1]

running -1 [930100006]

1612873812 :: Sending eventids [930100006] with specifier None

930100006 message is {sensor} out of spec in chassis {chassis} slot {slot}.

1.200 (70368744177864) corresponds to eventid 930100006

Out of events to run. Exiting.

# /usr/bin/isi_celog/celog_send_events.py -o 930100005

The output is:

Heap looptimes: [-1]

running -1 [930100005]

1612873817 :: Sending eventids [930100005] with specifier None

930100005 message is {sensor} out of spec in chassis {chassis} slot {slot}.

1.201 (70368744177865) corresponds to eventid 930100005

Out of events to run. Exiting.

3. To list all the events in the suppressed list, use the following command:

# isi event suppress list

The output is:

ID        Name

----------------------------

930100005 HWMON_ANY_DISCRETE

930100006 HWMON_ANY_METERS

----------------------------

4. These suppressed events will only show in the event history and will not trigger any notification in any channels.

Figure 14.    Suppressed events

5. Un-suppress the event types by clicking the Un-supress button.

Figure 15.    Un-suppress events

The following sections will demonstrate how to configure alerts based on several typical scenarios, including:

• Alert configurations for SyncIQ
• Alert configurations for disk rebuilds

Alert configurations for SyncIQ

Table 4 lists the key alerts for SyncIQ. Use the following event groups for alert subscription:

Table 4.      Event group for SyncIQ

For detailed information about each event group listed in Table 4, see the OneFS Event Reference.

Use the following command to subscribe to all the SyncIQ related events:

 isi event alerts create sync_iq_alert --eventgroup "synciq,400040012,400040020,400040021" NEW mychannel

Alert configurations for disk rebuilds

It is a very common scenario to get notified for each stage during a disk rebuild life cycle: when the disk fails, when the Flex Protect job completes, when the disk is back to a healthy status, and so on.

Table 5 lists the key stages, the corresponding event group id, and the disk state during the disk rebuild life cycle.

Table 5.      Event group configuration and disk state during a disk rebuild

Dell Technologies recommends using the new events as the condition of the alert configuration in order to get all the alert notifications of the entire disk rebuild stage as listed in Table 5. This configuration will ensure that you receive all events notifications including new and resolved for the subscribed event groups.

SMTP is the most commonly configured alerting protocol. The following sections will explain how SMTP alerts work in OneFS as well as configuration considerations and best practices.

How SMTP alerts work

Figure 16 shows the high-level architecture of SMTP alerts. As explained in the Alert architecture, OneFS sends an SMTP message when cluster events occur, sending them through configured SMTP channels to the SMTP server configured. The SMTP server will forward the email and attachments to the subscriber email address.

Figure 16.    SMTP alerts architecture

In the message body of SMTP Alerts, the following key information will be included as listed in Table 6:

Table 6.      Details in the message body of SMTP alerts

Configuration considerations

The general configuration for SMTP alerting requires the end user to provide the following information:

• Send from: The email address that you want your notifications to be sent from. This does not need to be a real account on a mail server unless required by your SMTP relay.
• Subject: This is the base subject that you want your notifications to have. Additional information about the notification will be added to the subject depending on the type of event and severity of the event.
• SMTP Host or relay address: This should be the public IP address or DNS name for the mail server which will accept SMTP relaying from your cluster.
• SMTP relay port: The port that your mail server will accept SMTP relaying over, typically this will be either port 25, or 465 if your mail server supports SSL for SMTP.
• Use SMTP authentication: It is optional to enable SMTP Authentication. Once it is enabled, the user information will be used to authenticate to the mail server.
• Notification batch mode: Enabling this option will group notifications together by severity, category, or you can have it batch all notifications together so that you only receive a single notification per event.
• Notification Email template: Here you can select a notification template file from your OneFS cluster to use for notifications.

PowerScale OneFS supports using public SMTP servers for alert notification, but it does not support multifactor authentication or 2-step authentication. For details, see the knowledge base article: Email notifications not working through smtp.gmail.com.

SMTP alerts support encrypted communication between SMTP servers and PowerScale clusters which is a more secure and recommended way for SMTP alert configuration. However, in some releases of OneFS version 8, when the encryption is configured in the channel, SMTP event notification will fail. We suggest you open a ticket with the support center to resolve this issue. For detailed information, refer to SMTP event notification was failed when configured Alert channel manually with STARTTLS for connection security.

SNMP alerts provide a standardized interface to query network devices for information. This can include uptime, descriptions, locations, and device-specific information. For OneFS, the device specific information includes cluster and node data, disk health, and much more.

There are two parts to the OneFS implementation of SNMP as listed below:

• SNMP monitoring: A synchronous way for administrators to query PowerScale information, usually for monitoring purposes.
• SNMP TRAP: An asynchronous way for administrators to subscribe to a subset of events.

This section will only cover SNMP TRAP. The SNMP monitoring will be discussed in section 2.

Note: At the time of writing, any value in the OneFS SNMP TRAP configuration cannot be customized.

How SNMP alerts work

As explained in the Alert architecture section, OneFS sends an SNMP TRAP when CELOG events occur, sending them through configured SNMP TRAP channels. Internally SNMP TRAPs are sent by process snmpinform or snmptrap with appropriate parameters to a third-party SNMP management console on the client side.

Figure 17.    SNMP TRAP architecture

As shown in this figure, either ‘snmpinform’ or ‘snmptrap’ can send the SNMP TRAP notification. The difference is that snmptrap is used to send the unacknowledged SNMP TRAP, which means that this process does not know whether the remote subscriber has received the message. The snmpinform, on the other hand, can send an acknowledged SNMP TRAP, which is also known as SNMP INFORM. For the detailed configuration of snmpinform and snmptrap, see the section Configuration considerations.

OneFS uses a custom management information base (MIB) to define human-readable names for managed SNMP objects and specify their data types and other properties. You can download these PowerScale specific MIBs from aa PowerScale node under /usr/share/snmp/mib/. For detailed information about MIB, see the section SNMP monitoring architecture.

Configuration considerations

SNMPv2c TRAP

Prior to OneFS 8.2.0, PowerScale OneFS only supports SNMPv2c TRAP, which means there is no authentication or encryption feature for SNMP TRAP. For the detailed difference between SNMPv2c and SNMPv3, see the section SNMP monitoring architecture.

To create an SNMP channel using snmptrap to send unacknowledged SNMP TRAP, use the following CLI command:

isi event channels create channel_1 snmp –-use-snmp-trap true

To create an SNMP channel using snmpinform to send acknowledgeable SNMP INFORMs, use the following CLI command:

isi event channels create channel_2 snmp –-use-snmp-trap false

Note: The switch to use snmpinform or snmptrap is only available in the CLI or PAPI (Platform API).

From OneFS version 8.0 onwards, CELOG uses snmpinform by default to send acknowledgeable SNMP TRAPs and sometimes this may not be supported in all customer environments. In these cases, it is recommended to modify the alert channel back to snmptrap. For details, see the KB article Upgrade to 8.0.x changes default SNMP behavior from 'snmptraps' to 'snmpinform'.

When an event channel has been set up to enable the snmptrap settings, changing the channel by the WebUI will lose the value of the snmptrap settings and revert to snmpinform settings. This issue is fixed in OneFS 8.0.0.6 and for the detailed information, see the KB article CELOGv2: Modify SNMP channel via WebUI reset use-snmp-trap.

The general configuration for SNMP alerts requires the end user to provide the following information:

• Community: The community name to subscribe the SNMP alert. The default value is public. The SNMP management software will check the community string extracted from an incoming SNMP TRAP request header to see if it matches the local configuration.
• Host: The host to receive SNMP alert

The following command is an example to create an SNMP channel using snmptrap to the public community on a host with IP address 10.xxx.xxx.xxx

isi event channels create snmpchannel snmp –-use-snmp-trap true –-community public –-host 10.xxx.xxx.xxx

After you have created the SNMP channel, create an alert to use this channel:

isi event alerts modify myalert --channel snmpchannel

It is recommended to test the SNMP alert settings before you configure the third-party SNMP management software. The following example uses snmptrapd to verify the configuration and test the SNMP alert on a remote server.

1. Copy ISILON-MIB.txt and ISILON-TRAP-MIB.txt from PowerScale to the directory of /usr/share/snmp/mibs/ in the remote machine.

2. Create a configuration file for snmptrapd. In this file, specify the community name as public.

echo "authCommunity log public" > /tmp/traps.cfg

3. Set up a minimal SNMP TRAP server using snmptrapd utility from the Net-SNMP toolchain.

snmptrapd -Lf /tmp/snmptrapd_traps.log -C -c /tmp/traps.cfg -p /tmp/at_snmp.pid -m ALL

4. Send a test alert and wait several minutes to check if there is a corresponding update in the /tmp/snmptrapd_traps.log file:

NET-SNMP version 5.7.2

2018-05-07 04:56:46 <UNKNOWN> [UDP: [10.yyy.yyy.yyy]:35488->[10.xxx.xxx.xxx]:162]:

DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (873813181) 101 days, 3:15:31.81             

SNMPv2-MIB::snmpTrapOID.0 = OID: ISILON-TRAP-MIB::testEventCrit             

ISILON-TRAP-MIB::instanceIdentifier = STRING: "2221" 

ISILON-MIB::clusterName = STRING: x41040g 

ISILON-MIB::nodeName = STRING: x41040g-1 

ISILON-MIB::nodeSerialNumber = STRING: SX410-301448-0070 

ISILON-TRAP-MIB::eventKbUrl = STRING: "Unavailable." 

ISILON-TRAP-MIB::eventKbUrl = STRING: "Unavailable."

SNMPv3 TRAP

Starting from OneFS 8.2.0, PowerScale supports SNMPv3 TRAP which provides more security options like authentication and encryption. The detailed security levels supported are listed below:

Table 7.      Supported security levels

The security level can be configured through the parameter --snmp-security-level under isi event channel modify. Unlike SNMPv2c, which use SNMP community string to identify the subscriber, SNMPv3 use MD5 or SHA encrypted passphrase for this purpose. Based on this, the community string should be left empty when the SNMPv3 channel is created. To set the authentication algorithm, use the parameter --snmp-auth-protocol and at the same time set the passphrase by --snmp-auth-password.

The SNMPv3 channel can only be created or configured through OneFS CLI. The following is an example to walk through all the configurations for this purpose. In this example, we create an SNMPv3 user - traptest whose passphrase is mypassword for both authentication (SHA) and encryption (AES). This user will be used in this SNMPv3 channel for the remote SNMPv3 host 10.7.xxx.xxx to subscribe the SNMPv3 TRAP.

1. Use the following CLI to create an SNMPv3 channel

isi event channels create snmpchannel snmp

--host 10.7.xxx.xxx

--use-snmp-trap True

--snmp-use-v3 True

--snmp-auth-protocol SHA

--snmp-auth-password mypassword

--snmp-priv-protocol AES

--snmp-priv-password mypassword

--snmp-security-level authPriv

--snmp-security-name traptest

--snmp-engine-id 0x8000000001020304

2. Use the following CLI to create an alert associated with this channel:

isi event alerts modify myalert --channel snmpchannel

It is recommended to test the SNMP alert settings before you configure the third-party SNMP management software. The following example uses snmptrapd to verify the configuration and test the SNMP alert on a remote server.

3. Copy ISILON-MIB.txt and ISILON-TRAP-MIB.txt from PowerScale to the directory of /usr/share/snmp/mibs/ in the remote machine.
4. Create a configuration file for snmptrapd. In this file, specify the SNMPv3 user information:

echo "createUser -e 0x8000000001020304 traptest SHA mypassword AES mypassword

authuser log traptest" > /tmp/traps.cfg

5. Set up a minimal SNMP TRAP server using snmptrapd utility from the Net-SNMP toolchain.

snmptrapd -Lf /tmp/snmptrapd_traps.log -C -c /tmp/traps.cfg -p /tmp/at_snmp.pid -m ALL

6. Send a test alert and wait several minutes to check if there is a corresponding update in the /tmp/snmptrapd_traps.log file:

DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (6229647) 17:18:16.47  SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-SMI::enterprises.12124.250.24.6.3  SNMPv2-SMI::enterprises.12124.250.50.15 = STRING: "9"   

SNMPv2-SMI::enterprises.12124.1.1.1 = STRING: "vshen-5coqidt"  

SNMPv2-SMI::enterprises.12124.2.1.1 = STRING: "vshen-5coqidt-1"     SNMPv2-SMI::enterprises.12124.2.1.5 = STRING: "SV200-004EIJ-5XA4"       SNMPv2-SMI::enterprises.12124.250.50.19 = STRING: "400150007"       SNMPv2-SMI::enterprises.12124.250.50.20 = STRING: "SW_UPGRADE_NODE_NON_RESPONSIVE"      SNMPv2-SMI::enterprises.12124.250.50.18 = STRING: http://doc.isilon.com/onefs/8.2/help/en-us/#ifs_r_event_400150007.html

Clear SNMP

Clear SNMP TRAP is introduced in OneFS 8.2.0. When an issue happens and is detected by CELOG, an SNMP TRAP will be sent if the corresponding SNMP channel has been created before. After this issue is fixed and the event is marked resolved, a clear SNMP TRAP will automatically be sent to the SNMP subscriber. The overall workflow is shown in Figure 18.

Figure 18.    Clear SNMP TRAP

In the example shown in this figure, an issue happens due to the missing mirror boot disk. This issue is detected by the CELOG, and an SNMP TRAP is delivered with OID: bootDiskMirrorReqMissingCrit to the SNMP subscriber. After this issue is fixed and the event group is marked as resolved, a clear SNMP TRAP is delivered automatically to notify to the subscriber. The OID, in this case, is bootDiskMirrorReqMissingCritClear.

Note: From OneFS 8.2.0, all SNMP TRAP should have clear versions.

CELOG and SNMP TRAPS

From OneFS 8.2.0, All the CELOG eventgroups now have associated SNMP TRAP.

You can use SNMP to remotely monitor the PowerScale cluster hardware components, such as fans, hardware sensors, power supplies, and disks. Use the default Linux® SNMP tools or a UI-based SNMP tool of your choice for this purpose.

Note: SNMP monitoring and SNMP TRAP are two different topics. SNMP monitoring is the query to get the value of the SNMP entities by SNMP GET in a synchronized way. SNMP TRAP is used to subscribe the events by SNMP INFORM/TRAP in an asynchronous way.

OneFS SNMP monitoring feature only supports SNMPv2c and SNMPv3. Compared with SNMPv2c, SNMPv3 adds both authentication and encryption features. Table 8 lists the supportability for OneFS SNMP monitoring.

Table 8.      SNMPv2c and SNMPv3 supportability

The architecture of the SNMP monitoring feature is shown in Figure 19:

Figure 19.    SNMP monitoring architecture

SNMP applications run in a network management system (NMS) and issue queries (SNMP GET/GET NEXT) to the SNMP service on PowerScale to gather information. snmpd – the SNMP daemon on the cluster responds to the queries and sends the corresponding statistics to the SNMP applications. An SNMP community is a logical relationship between the SNMP service on the OneFS side and the NMS on the client side. The community has a name, and the default name for PowerScale OneFS is I$ilonpublic.

Elements in an SNMP hierarchy are arranged in a tree structure, similar to a directory tree. As with directories, identifiers move from general to specific as the string progresses from left to right. Unlike a file hierarchy, each element is not only named, but also numbered.

For example, the SNMP entity .iso.org.dod.internet.private.enterprises.isilon.oneFSss.ssLocalNodeId maps to the object id (OID) .1.3.6.1.4.1.12124.3.2 as shown in Figure 20. The part of the name that refers to the OneFS SNMP namespace is the 12124.3 element. Anything further to the right of that number is related to OneFS specific monitoring.

Management Information Base (MIB) documents define human-readable names for managed objects and specify their data types and other properties. You can download MIBs that are created for SNMP monitoring of aa PowerScale cluster from the OneFS web administration interface or manage them using CLI. MIBs are stored in /usr/share/snmp/mibs/ on a OneFS node. The OneFS MIBs serve two purposes:

• Provide automatic name to OID mapping
• Provide OneFS specific information that is unavailable in standard MIBs

A PowerScale cluster has two separate MIBs:

• ISILON-MIB is used for the SNMP monitoring feature.
• ISILON-TRAP-MIB is used for the SNMP TRAP feature.

Figure 20.    PowerScale MIB example

The following two sections will provide some guidelines and configuration considerations on the SNMP monitoring feature in OneFS.

• General considerations: Introduction and step-by-step procedure on how to configure SNMP monitoring
• Security considerations: Introduction of security best practices dealing with SNMP monitoring

General considerations

By default, the SNMP monitoring feature is enabled and can be configured from both the CLI (isi snmp settings modify) and the web UI (SNMP Monitoring under General Settings of Cluster Management). You can enable or disable SNMP monitoring, allow SNMP access by version, and configure other settings, some of which are optional. All SNMP access is read-only.

There is one major difference between the CLI and the web UI. In CLI, it allows you to send the passphrase for encryption through the parameter –set-snmp-v3-priv-password, meanwhile there is no such option in the web UI. The recommendation is to not modify this passphrase. In case you modified this and met with an error, open a service request with Dell Support.

For step-by-step configuration details for the SNMP monitoring feature, see the CLI Administration Guide.

After the configuration is completed, you can use snmpwalk to validate if the configuration is good or not. This tool can be run either on a remote host or on a PowerScale node itself.

Note: This tool is pre-installed on the PowerScale system.

For SNMPv2c, the following is an example of using local snmpwalk running on the PowerScale to validate the configuration.

f800eth-1# snmpwalk -v2c -c I\$ilonpublic -m all localhost ifsFilesystem

ISILON-MIB::ifsTotalBytes.0 = Counter64: 189940499251200

ISILON-MIB::ifsUsedBytes.0 = Counter64: 22635765792768

ISILON-MIB::ifsAvailableBytes.0 = Counter64: 160600454995968

ISILON-MIB::ifsFreeBytes.0 = Counter64: 167304733458432

ISILON-MIB::ifsFilesystem.5.0 = INTEGER: 0

ISILON-MIB::ifsFilesystem.6.0 = Gauge32: 86400000

For SNMPv3, use the following example to validate the configuration.

snmpwalk -m all -c I\$ilonpublic -v 3 -l authnopriv -u general -a md5 -A 'password' localhost ifsFilesystem

ISILON-MIB::ifsTotalBytes.0 = Counter64: 400611948904448

ISILON-MIB::ifsUsedBytes.0 = Counter64: 10623134490624

ISILON-MIB::ifsAvailableBytes.0 = Counter64: 381578066165760

ISILON-MIB::ifsFreeBytes.0 = Counter64: 389988814413824

ISILON-MIB::ifsFilesystem.5.0 = INTEGER: 0

ISILON-MIB::ifsFilesystem.6.0 = Gauge32: 86400000

For details on how to use snmpwalk, see the section snmpwalk.

Security considerations

If you plan to monitor cluster statistics, SNMPv3 is recommended. When SNMPv3 is used, OneFS requires the SNMP-specific security level of AuthNoPriv as the default value when querying the PowerScale cluster. At the same time, it is recommended to disable SNMPv1 and SNMPv2c access.

Note: OneFS does not support SNMPv1. Although an option for –snmp-v1-v2c-access exists in the OneFS CLI command. If you turn on that parameter, OneFS will only monitor through SNMPv2c.

To do this, use the following CLI command line:

1. Enable SNMPv3 access:

isi snmp settings modify –snmp-v3-access=yes

2. Disable SNMPv1 and SNMPv2c access

isi snmp settings modify –snmp-v1-v2c-access=no

For security, it is recommended to disable the SNMP service if SNMP monitoring is not required. Disabling SNMP on the cluster does not affect the sending of SNMP TRAP alerts from the cluster to an SNMP management server.

To disable the SNMP monitoring feature in OneFS, use the following CLI command line:

isi services snmp disable

This section describes tools which can be used to validate SNMP TRAP or SNMP monitoring configurations. Also, there are some useful OneFS CLI commands which will be discussed in this section. The common CLI commands to create and configure alerts and channels will not be covered in this section. For that part, see the OneFS CLI Administration Guide.

snmpwalk is the tool in the Net-SNMP tool chain to retrieve or query the SNMP entities using SNMP GET request. This tool is also pre-installed on the cluster, making it easy to use if you want to validate the SNMP monitoring configuration locally instead of on a different host. snmpwalk supports both SNMPv2c and SNMPv3.

For SNMPv2c, use the following example to validate the configuration.

f800eth-1# snmpwalk -v2c -c I\$ilonpublic -m all localhost ifsFilesystem

ISILON-MIB::ifsTotalBytes.0 = Counter64: 189940499251200

ISILON-MIB::ifsUsedBytes.0 = Counter64: 22635765792768

ISILON-MIB::ifsAvailableBytes.0 = Counter64: 160600454995968

ISILON-MIB::ifsFreeBytes.0 = Counter64: 167304733458432

ISILON-MIB::ifsFilesystem.5.0 = INTEGER: 0

ISILON-MIB::ifsFilesystem.6.0 = Gauge32: 86400000

The above command means to query the SNMP entity – “ifsFilesystem” on this PowerScale node under the community of “I\$ilonpublic” using SNMPv2c protocol.

For SNMPv3, use the following example to validate the configuration.

snmpwalk -m all -c I\$ilonpublic -v 3 -l authnopriv -u general -a md5 -A 'password' localhost ifsFilesystem

ISILON-MIB::ifsTotalBytes.0 = Counter64: 400611948904448

ISILON-MIB::ifsUsedBytes.0 = Counter64: 10623134490624

ISILON-MIB::ifsAvailableBytes.0 = Counter64: 381578066165760

ISILON-MIB::ifsFreeBytes.0 = Counter64: 389988814413824

ISILON-MIB::ifsFilesystem.5.0 = INTEGER: 0

ISILON-MIB::ifsFilesystem.6.0 = Gauge32: 86400000

snmptrapd is the tool in the Net-SNMP tool chain that receives and logs SNMP TRAP messages. By default, it listens on UDP port 162 on all IPv4 interfaces. Since port 162 is a privileged port, snmptrapd must be run as root.

To set up a minimal SNMP TRAP server to validate the OneFS SNMP TRAP settings, use the following CLI commands.

1. Create a temporary configuration file for snmptrapd to log messages for the public community. The community string should be aligned with the configuration in the SNMP channel. For snmptrapd configuration file details, see the snmptrapd.conf(5) - Linux man page.

echo "authCommunity log public" > /tmp/traps.cfg

2. Start the snmptrapd daemon services to receive the SNMP TRAP. The parameter “-Lf /tmp/snmptrapd_traps.log” means to log all incoming messages to a specific file under /tmp. The parameter “-C -c /tmp/traps.cfg” means not to read the default configuration files, but to read the file which was created in the first step for configuration. For snmptrapd details, see SNMPTRAPD.

snmptrapd -Lf /tmp/snmptrapd_traps.log -C -c /tmp/traps.cfg -m ALL

3. Set up the SNMP channel and alert in OneFS and send a test alert.

Wait for several minutes, then check to see if there is a corresponding entry in the /tmp/snmptrapd_traps.log file.

NET-SNMP version 5.7.2

2018-05-07 04:56:46 <UNKNOWN> [UDP: [10.yyy.yyy.yyy]:35488->[10.xxx.xxx.xxx]:162]:

DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (873813181) 101 days, 3:15:31.81             

SNMPv2-MIB::snmpTrapOID.0 = OID: ISILON-TRAP-MIB::testEventCrit             

ISILON-TRAP-MIB::instanceIdentifier = STRING: "2221" 

ISILON-MIB::clusterName = STRING: x41040g 

ISILON-MIB::nodeName = STRING: x41040g-1 

ISILON-MIB::nodeSerialNumber = STRING: SX410-301448-0070 

ISILON-TRAP-MIB::eventKbUrl = STRING: "Unavailable." 

ISILON-TRAP-MIB::eventKbUrl = STRING: "Unavailable."

To validate the OneFS SNMP TRAP configuration in a Windows environment, see the KB article How to install 'Net-SNMP' tools in Windows and use snmptrapd/snmpwalk to test receive/send SNMP trap information.

This section lists some useful CLI commands to configure alerts in OneFS:

Check SNMP monitoring service

Use the following CLI command to check if SNMP monitoring service is enabled on the PowerScale node:

isi services -a snmp

To check this service for all the PowerScale nodes, use the following script:

isi_for_array -sX 'pgrep snmpd | wc -l'

If the service is enabled on all the PowerScale nodes, the output will look like this:

f800eth-1:        1

f800eth-2:        1

f800eth-3:        1

f800eth-4:        1

Check port connectivity

OneFS SNMP monitoring service and SNMP TRAP use dedicated ports for external communications as listed in Table 9.

Table 9.      SNMP monitoring and SNMP TRAP connectivity

Use the following command to query the port connectivity for snmp or snmptrap on a dedicated PowerScale node

x41040g-1# nc -u -z localhost 161

Connection to localhost 161 port [udp/snmp] succeeded!

x41040g-1# nc -u -z localhost 162

Connection to localhost 162 port [udp/snmptrap] succeeded!

To check the port connectivity for all the PowerScale nodes, use the following script:

isi_for_array -sX 'nc -u -z localhost 161'

If the port is reachable on all the PowerScale nodes, the output will look like this:

x41040g-1: Connection to localhost 161 port [udp/snmp] succeeded!

x41040g-2: Connection to localhost 161 port [udp/snmp] succeeded!

x41040g-3: Connection to localhost 161 port [udp/snmp] succeeded!

isi event types

From OneFS 8.0.0.5 onwards, the CLI – isi event types is introduced and will list all the supported event types for SMTP alert which can be configured in the OneFS alert configuration. This is useful in a scenario such as disabling certain event types for alert configuration. For details, see the section Alert configuration by the deny list.

To list all the supported event types in csv format, use the following CLI command:

isi event types list –format=csv

Note: For all event groups which can be subscribed through SNMP, refer to appendix A.

Table 10 lists all the ISILON-MIB details for the SNMP monitoring feature.

Table 10.  SNMP monitoring and SNMP TRAP connectivity

Table 11.  Full list of SRS brevity

The following Dell Technologies documentation provides other information related to this document. Access to these documents depends on your login credentials. If you do not have access to a document, contact your Dell Technologies representative.

• OneFS CLI Administration Guide
• PowerScale Troubleshooting Guide: Administration - SNMP
• OneFS Security Configuration Guide
• OneFS Event Reference