PowerFlex How to check size of and clear Elasticsearch data vxfm00513

ERROR http://localhost:9200/snmp-traps-YYYY.MM.DD/<Serial>/_update
ERROR,HTTP error from ElasticSearch: 403 Client Error: Forbidden for url: http://localhost:9200/snmp-traps-2020.07.20/snmptraps/UgreanMBUFFdoaMHXN_S/_update

• Space issues in PowerFlex Manager on /var
• Elastic search database is growing too large (ex. larger environment with more alerts)
• Debug why elastic search is having issues:
  • Check logs
  • Check space on /var 

*Note: after each step listed, re-run inventory or retest alerts (send test alert). If there is still an issue, move on to the next step.


Step 1 = free up space in elastic search database

• Check the size of elastic search indexes on disk: 
  •    

    curl 'localhost:9200/_cat/indices?v’

  •    Note: The max size that it should be is 5G
    • If running PFxM 3.4.x or earlier, the size can grow larger
    • Recommend upgrading to PFxM 3.6.x

• **syslog indices are safe to delete from elastic search db
  • curl -X DELETE 'http://localhost:9200/syslog*'
  • Retry test alerts

• If a customer has a large number of snmp trap indices, it is ok to delete older traps if the customer is ok with it:

  • curl -X DELETE localhost:9200/snmp-traps-YYYY.MM.DD (this deletes a specific date)

Step 2 = remove elastic search logs.

• sudo systemctl status Elasticsearch
• sudo systemctl stop Elasticsearch
•  cd /var/log/elasticsearch (note: Do not delete vxfm-es-cluster.log)

  • rm -rf *log.gz*
    rm -rf *gc.log*
    rm -rf *showlog.log*

• Systemctl start Elasticsearch
• sudo systemctl status Elasticsearch

Step 3 = Delete ALL data in ElasticSearch: 

*Warning: The customer will lose all historical alerts, performance metrics from resources, and syslog data. Ensure that the customer is ok with losing this data prior to running this command.

curl -XDELETE localhost:9200/_all

 
Step 4 = grow PowerFlex Manager Filesystem = How to grow a partition on a VM running Logical Volume Manager (LVM)
Note: grow PFxM appliance on vm too (edit settings, grow hard disk by amount of extra space added)

If there are still issue, contact support.

Note: elastic search database lives here ' do not delete anything from this directory: /var/lib/elasticsearch/
Note: elastic search database contains alerts and metric data for PowerFlex and some syslog

Notes: 

• PowerFlex Manager 3.7 version has a larger file system
• As of PowerFlex Manager 3.6, logs are rotated changed from rotating every day to every 15 minutes.
• Patch has been created for to rotate logs every 15 minutes for PowerFlex Manager 3.4 & 3.5 (fixed in 3.6…so upgrade!)
• If the index exceeds the values below, it rotates (delete older entries)
  • Metrics (PowerFlex, Nodes, and Cisco 9k Switches: 30 GB (15 GB each)
  • SNMP Traps: 5 GB
  • Syslog: 5 GB

 See PowerFlex Upgrade from 3.x to 4.x fails a database backup fails due to issue with ElasticSearch