VMAX All Flash, PowerMax V3 & V4: AUDIT LOGS: All you must know.

Audit Log:

The Audit Log records major activities on an array, including: 

• Host-initiated actions 
• Physical component changes 
• Actions on the MMCS 
• D@RE key management events 
• Attempts blocked by security controls (Access Controls) 

The Audit Log is secure and tamper-proof so event contents cannot be altered. Users with Auditor access can view, but not modify, the log.

Storage system audit records come from the SYMAPI database and include all actions that are taken on that storage system. 

The Audit Log is stored on the PowerMax array. It is 1 GB in size.

The Audit Log message catalog contains audit messages generated by Solutions Enabler and recorded in the storage system. With PowerMaxOS 10, a new standardized audit format has been introduced. Also, for storage systems running HYPERMAX OS 5977 or PowerMaxOS 5978, users may select the new format instead of the legacy format.

The Audit Log: 

• Has advanced search and filter capabilities, such as: 
  • Specific time period (for example, last 24 hours, last week, specific date) 
  • Specific user (username, user role) 
  • Operation type and category 
  • Application type (Unisphere, CLI) 
  • Hostname 
  • Search phrase 
• Records and the filtered Audit Log list can be exported to an .log file. This is facilitated with a REST endpoint.

AUDIT LOG vs AUDIT EVENTS

Audit log
The Audit logs are an internal record to keep almost everything that happens on the array. You can use Solutions Enabler to query the Audit Log. symaudit list is the command with various options. This is the log that is on the array and is 1GB in size and holds millions of records. This is not to be confused with audit events.

Audit events
Event daemon can be used to report on a few audit events. These are listed in the events and alerts guide. Search on "Audit" and find the small number of events that are focused on the audit log. For example:

Event 1403
This is an informational event triggered when the event daemon encounters a 1403. This can correspond to anything listed in the notes section provided the user has configured it in the event daemon. If the event daemon is configured for syslog, then these small number of events can be directed to that syslog.

symaudit -sid <sid> show is useful to quickly see how far back the Audit Log goes and to get a record count.

Example from lab of 5978 and 6079 arrays:

root@testlab ~]# symaudit show -sid 111
 
              A U D I T   L O G   D A T A
 
Symmetrix ID            : 000197700111

Starting date           : 12/04/2018 13:14:12
Ending date             : 04/30/2025 02:48:27

Starting record number  :       1
Ending record number    : 1704552
Total record count      : 1704552
Audit Format            :  Legacy

[root@testlab ~]# symaudit show -sid 222

              A U D I T   L O G   D A T A

Symmetrix ID            : 000120002222

Starting date           : 02/22/2023 15:15:22
Ending date             : 04/30/2025 02:46:41

Starting record number  :       1
Ending record number    :  303868
Total record count      :  303868
Audit Format            :     New

The starting record number of 1 implies that the Audit Log on those arrays has not yet wrapped.
 
Example of a wrapped Audit Log:  

[root@testlab ~]# symaudit show -sid 333

              A U D I T   L O G   D A T A

Symmetrix ID            : 000197600333

Starting date           : 11/18/2022 04:42:26
Ending date             : 04/30/2025 03:04:18

Starting record number  : 2390445
Ending record number    : 4487596
Total record count      : 2097152
Audit Format            :  Legacy

For that wrapped log, any record previous to 2390445 is no longer available.

Where is this Audit file physically located, path and filename?

• An Audit Log is not a standard log file stored in a fixed location. Instead, it is generated using specific symaudit commands. Audit data is retrieved from the Symmetrix file system in older arrays or from Global Memory in newer ones, then manually copied into a text file.
• We cannot see the audit file as it is not in a location we can access.

Can the 1 GB limited file can be editable to change the file size (so Audit logs can be written more)?
We cannot alter the 1 GB size. It is stored in a buffer on the array. When a symaudit list command is issued, the syscalls are passed to the array and the display appears.

If the 1 GB size cannot alter, how many months or years would it take for the 1 GB size to be overwritten?
For new arrays, the size of an audit log is 1GB in total, that is, room for about 2 million records. Each record is a fixed size, 512 bytes. This is the case for V4 arrays as well.

When the log is full, it wraps and starts overwriting the oldest records. However, 2 million records are a lot. This seldom happens.

The Event Daemon exposes an event that says it is getting close to wrapping. The idea is that customers could take whatever action they want. For example, exporting the log to some database so that nothing is lost.

How quickly it wraps depends on how much activity there is, and how quickly audit records are being written.

If one record is written every minute, it will wrap in about four years.

With steady background orchestration and other activity (such as VASA), it may write records more often than that. That has always been a source of conflicting goals. We want to write everything that happens, but we also do not want to wrap the Audit Log too often.

The following are challenges to increasing the Audit Log size:

1. It is expensive since it is held in global memory on the array.
2. Doubling the size did not immediately seem like it would have a significant impact. Wrap could still occur, so everyone must be ready for that possibility. The key question is how frequently it happens. We cannot make it infinitely large or ensure a wrap-free experience.

Can we forward Audit logs to an External Syslog server?

No, we cannot.

Could Unisphere for PowerMax experience performance slowdowns as the logs become full?

No, there is no impact.

Is there a preferred retention that can be set?

No, the 1 GB size cannot be altered.

For older arrays such as 100K, 200K, 400K the following explanation applies:

Data is written to a common audit file during VMAX control operations. The common Audit Log correlates activity from all hosts into one file stored in the Symmetrix File System (SFS). The symaudit command enables the filtering of the common Audit Log file for a specified VMAX array. The Audit Log resides on the array with a capacity of 40 MB. Once the 40 MB limit is reached, the log starts to overwrite itself. There is no maintenance for this file, unless records are captured before the circular 40 MB space recycles.