ECS 3.5.0.1 Administration Guide

Database monitoring_main

Performance metrics in this database are raw, each is split by data node, that is all have host and node_id tags.

Information:
Measurement in this section have following structure:

service_IO_Statistics_data_read - for read I/O counters

service_IO_Statistics_data_write - for read I/O counters

Service is the name of ECS service that produces the measurement, i.e. blob, cm, georcv, statDataHead.

For example,

	blob_IO_Statistics_data_read
	cm_IO_Statistics_data_write

Measurement: blob_IO_Statistics_data_read
             ...
Tags: host, node_id, process, tag
Fields: read_CCTotal (float, bytes)
        read_ECTotal (float, bytes)
        read_GEOTotal (float, bytes)
        read_RECOVERTotal (float, bytes)
        read_USERTotal (float, bytes)
        read_XORTotal (float, bytes)
 
Measurement: blob_IO_Statistics_data_write
             ...
Tags: host, node_id, process, tag
Fields: write_CCTotal (integer)
        write_ECTotal (integer)
        write_GEOTotal (integer)
        write_RECOVERTotal (integer)
        write_USERTotal (integer)
        write_XORTotal (integer)

Measurement: blob_SSDReadCache_Stats
Tags: host, id, last, node_id, process
Fields: +Inf (integer)
        0.0 (integer)
        1000.0 (integer)
        25000.0 (integer)
        5000.0 (integer)
        rocksdb_disk_capacity_failure_counter (integer)
        rocksdb_disk_usage_counter_bytes (integer)
        rocksdb_disk_usage_percentage_counter (integer)
        ssd_capacity_counter_bytes (integer)

These statistics represent processes in ECS service CM, such BTree GC, Chunk management, Erasure coding.

Measurement: cm_BTREE_GC_Statistics
Tags: host, node_id, process, tag
Fields: accumulated_candidate_garbage_btree_gc_level_0 (integer)
        accumulated_candidate_garbage_btree_gc_level_1 (integer)
        accumulated_detected_data_btree_level_0 (integer)
        accumulated_detected_data_btree_level_1 (integer)
        accumulated_reclaimed_data_btree_level_0 (integer)
        accumulated_reclaimed_data_btree_level_1 (integer)
        candidate_chunks_btree_gc_level_0 (integer)
        candidate_chunks_btree_gc_level_1 (integer)
        candidate_garbage_btree_gc_level_0 (integer)
        candidate_garbage_btree_gc_level_1 (integer)
        copy_candidate_chunks_btree_gc_level_0 (integer)
        copy_candidate_chunks_btree_gc_level_1 (integer)
        copy_completed_chunks_btree_gc_level_0 (integer)
        copy_completed_chunks_btree_gc_level_1 (integer)
        copy_waiting_chunks_btree_gc_level_0 (integer)
        copy_waiting_chunks_btree_gc_level_1 (integer)
        deleted_chunks_btree_level_0 (integer)
        deleted_chunks_btree_level_1 (integer)
        deleted_data_btree_level_0 (integer)
        deleted_data_btree_level_1 (integer)
        full_reclaimable_chunks_btree_gc_level_0 (integer)
        full_reclaimable_chunks_btree_gc_level_1 (integer)
        reclaimed_data_btree_level_0 (integer)
        reclaimed_data_btree_level_1 (integer)
        usage_between_0%_and_5%_chunks_btree_gc_level_0 (integer)
        usage_between_0%_and_5%_chunks_btree_gc_level_1 (integer)
        usage_between_10%_and_15%_chunks_btree_gc_level_0 (integer)
        usage_between_10%_and_15%_chunks_btree_gc_level_1 (integer)
        usage_between_5%_and_10%_chunks_btree_gc_level_0 (integer)
        usage_between_5%_and_10%_chunks_btree_gc_level_1 (integer)
        verification_waiting_chunks_btree_gc_level_0 (integer)
        verification_waiting_chunks_btree_gc_level_1 (integer)
 
Measurement: cm_Chunk_Statistics
Tags: host, node_id, process, tag
Fields: chunks_copy (integer)
        chunks_copy_active (integer)
        chunks_copy_s0 (integer)
        chunks_level_0_btree (integer)
        chunks_level_0_btree_active (integer)
        chunks_level_0_btree_active_index_page (integer)
        chunks_level_0_btree_active_leaf_page (integer)
        chunks_level_0_btree_index_page (integer)
        chunks_level_0_btree_leaf_page (integer)
        chunks_level_0_btree_s0 (integer)
        chunks_level_0_btree_s0_index_page (integer)
        chunks_level_0_btree_s0_leaf_page (integer)
        chunks_level_0_journal (integer)
        chunks_level_0_journal_active (integer)
        chunks_level_0_journal_s0 (integer)
        chunks_level_1_btree (integer)
        chunks_level_1_btree_active (integer)
        chunks_level_1_btree_active_index_page (integer)
        chunks_level_1_btree_active_leaf_page (integer)
        chunks_level_1_btree_index_page (integer)
        chunks_level_1_btree_leaf_page (integer)
        chunks_level_1_btree_s0 (integer)
        chunks_level_1_btree_s0_index_page (integer)
        chunks_level_1_btree_s0_leaf_page (integer)
        chunks_level_1_journal (integer)
        chunks_level_1_journal_active (integer)
        chunks_level_1_journal_s0 (integer)
        chunks_repo (integer)
        chunks_repo_active (integer)
        chunks_repo_s0 (integer)
        chunks_typeII_ec_pending (integer)
        chunks_typeI_ec_pending (integer)
        chunks_undertransform_ec_pending (integer)
        chunks_xor (integer)
        data_copy (integer)
        data_level_0_btree (integer)
        data_level_0_btree_index_page (integer)
        data_level_0_btree_leaf_page (integer)
        data_level_0_journal (integer)
        data_level_1_btree (integer)
        data_level_1_btree_index_page (integer)
        data_level_1_btree_leaf_page (integer)
        data_level_1_journal (integer)
        data_repo (integer)
        data_repo_copy (integer)
        data_xor (integer)
        data_xor_shipped (integer)
 
Measurement: cm_EC_Statistics
Tags: host, node_id, process, tag
Fields: chunks_ec_encoded (integer)
        chunks_ec_encoded_alive (integer)
        data_ec_encoded (integer)
        data_ec_encoded_alive (integer)
 
Measurement: cm_Geo_Replication_Statistics_Geo_Chunk_Cache
Tags: host, node_id, process, tag
Fields: Capacity_of_Cache (integer)
        Number_of_Chunks (integer)
 
Measurement: cm_REPO_GC_Statistics
Tags: host, node_id, process, tag
Fields: accumulated_deleted_garbage_repo (integer)
        accumulated_reclaimed_garbage_repo (integer)
        deleted_chunks_repo (integer)
        deleted_data_repo (integer)
        ec_freed_slots (integer)
        full_reclaimable_aligned_chunk (integer)
        merge_copy_overhead_in_deleted_data_repo (integer)
        merge_copy_overhead_in_reclaimed_data_repo (integer)
        reclaimed_chunk_repo (integer)
        reclaimed_data_repo (integer)
        slots_waiting_shipping (integer)
        slots_waiting_verification (integer)
        total_ec_free_slots (integer)
 
Measurement: cm_Rebalance_Statistics
Tags: host, node_id, process, tag
Fields: bytes_rebalanced (integer)
        bytes_rebalancing_failed (integer)
        chunks_canceled (integer)
        chunks_for_rebalancing (integer)
        chunks_rebalanced (integer)
        chunks_total (integer)
        jobs_canceled (integer)
        segments_for_rebalancing (integer)
        segments_rebalanced (integer)
        segments_rebalancing_failed (integer)
        segments_total (integer)
 
Measurement: cm_Rebalance_Statistics_CoS
Tags: CoS, host, node_id, process, tag
Fields: bytes_rebalanced (integer)
        bytes_rebalancing_failed (integer)
        chunks_canceled (integer)
        chunks_for_rebalancing (integer)
        chunks_rebalanced (integer)
        chunks_total (integer)
        jobs_canceled (integer)
        segments_for_rebalancing (integer)
        segments_rebalanced (integer)
        segments_rebalancing_failed (integer)
        segments_total (integer)
 
Measurement: cm_Recover_Statistics
Tags: host, node_id, process, tag
Fields: chunks_to_recover (integer)
        data_recovered (integer)
        data_to_recover (integer)
 
Measurement: cm_Recover_Statistics_CoS
Tags: CoS, host, node_id, process, tag
Fields: chunks_to_recover (integer)
        data_recovered (integer)
        data_to_recover (integer)

These statistics represent processes in ECS service SR, responsible for space reclamation.

Measurement: sr_REPO_GC_Statistics
Tags: host, node_id, process, tag
Fields: accumulated_merge_copy_overhead_in_full_garbage (integer)
        accumulated_total_repo_garbage (integer)
        full_reclaimable_repo_chunk (integer)
        garbage_in_partial_sr_tasks (integer)
        garbage_in_repo_usage (integer)
        merge_copy_overhead_in_full_garbage (integer)
        merge_way_gc_processed_chunks (integer)
        merge_way_gc_src_chunks (integer)
        merge_way_gc_targeted_chunks (integer)
        merge_way_gc_tasks (integer)
        total_repo_garbage (integer)
        usage_between_0%_and_33.3%_repo_chunk (integer)
        usage_between_33.3%_and_50%_repo_chunk (integer)
        usage_between_50%_and_66.7%_repo_chunk (integer)

These statistics represent processes in ECS storage manager service SSM.

Measurement: ssm_sstable_SSTable_SS
Tags: SS, SSTable, last, process, tag
Fields: allocatedSpace (integer)
        availableFreeSpace (integer)
        downDurationTotal (integer)
        freeSpace (integer)
        largeBlockAllocated (integer)
        largeBlockAllocatedSize (integer)
        largeBlockFreed (integer)
        largeBlockFreedSize (integer)
        pendingDurationTotal (integer)
        pingerDurationTotal (integer)
        smallBlockAllocated (integer)
        smallBlockFreed (integer)
        smallBlockFreedSize (integer)
        smallBlockSize (integer)
        state (string)
        timeInStateTotal (integer)
        totalSpace (integer)
        upDurationTotal (integer)
 
Measurement: ssm_sstable_SSTable_SS_datamigration
Tags: SS, SSTable, last, process
Fields: status (integer)
        totalCapacityToMigrate (integer)

Database monitoring_last

Measurement: blob_Process_status
Tags: Process_status, host, node_id, process
Fields: MemoryTableFreeSpacePercentagePerMinute (integer)
        NumberofWritePageAllocationOutsideWriteCache (integer)
 
Measurement: blob_Total_memory_and_disk_cache_size
Tags: Total_memory_and_disk_cache_size, host, last, node_id, process
Fields: Disk_cache_size (integer)
        Memory_cache_size (integer)
 
Measurement: cm_Process_status
Tags: Process_status, host, node_id, process
Fields: MemoryTableFreeSpacePercentagePerMinute (integer)
        NumberofWritePageAllocationOutsideWriteCache (integer)
 
Measurement: eventsvc_Process_status
Tags: Process_status, host, node_id, process
Fields: MemoryTableFreeSpacePercentagePerMinute (integer)
        NumberofWritePageAllocationOutsideWriteCache (integer)
 
Measurement: mm_Process_status
Tags: Process_status, host, node_id, process
Fields: MemoryTableFreeSpacePercentagePerMinute (integer)
        NumberofWritePageAllocationOutsideWriteCache (integer)
 
Measurement: resource_Process_status
Tags: Process_status, host, node_id, process
Fields: MemoryTableFreeSpacePercentagePerMinute (integer)
        NumberofWritePageAllocationOutsideWriteCache (integer)
 
Measurement: rm_Process_status
Tags: Process_status, host, node_id, process
Fields: MemoryTableFreeSpacePercentagePerMinute (integer)
        NumberofWritePageAllocationOutsideWriteCache (integer)
 
Measurement: sr_Process_status
Tags: Process_status, host, node_id, process
Fields: MemoryTableFreeSpacePercentagePerMinute (integer)
        NumberofWritePageAllocationOutsideWriteCache (integer)
 
Measurement: sr_Total_memory_and_disk_cache_size
Tags: Total_memory_and_disk_cache_size, host, last, node_id, process
Fields: Disk_cache_size (integer)
        Memory_cache_size (integer)
 
Measurement: ssm_Process_status
Tags: Process_status, host, node_id, process
Fields: MemoryTableFreeSpacePercentagePerMinute (integer)
        NumberofWritePageAllocationOutsideWriteCache (integer)

Measurement: dtquery_cmf
Tags: last, process
Fields: com.emc.ecs.chunk.gc.btree.enabled (integer)
        com.emc.ecs.chunk.gc.btree.scanner.verification.enabled (integer)
        com.emc.ecs.chunk.gc.repo.enabled (integer)
        com.emc.ecs.chunk.gc.repo.verification.enabled (integer)
        com.emc.ecs.chunk.rebalance.is_enabled (integer)
        com.emc.ecs.objectgc.cas.enabled (integer)
        com.emc.ecs.sensor.btree_sr_pending_mininum (integer)
        com.emc.ecs.sensor.repo_sr_pending_mininum (integer)

Measurement: mm_topn_bucket_by_obj_count_place
Tags: last, place, process, tag
Fields: bucketName (string)
        namespace (string)
        value (integer)
 
Measurement: mm_topn_bucket_by_obj_size_place
Tags: last, place, process, tag
Fields: bucketName (string)
        namespace (string)
        value (integer)

Measurement: vnestStat_membership_ismember
Tags: host, ismember, last, node_id, process
Fields: is_leader (string)
 
Measurement: vnestStat_performance_latency_type
Tags: host, id, last, node_id, process, type
Fields: +Inf (integer)
        0.0 (integer)
        1.0 (integer)
        7999999.99999999 (integer)
        825912.9477680004 (integer)
        85266.52466135359 (integer)
        8802.840841123942 (integer)
        9.686250859269972 (integer)
        908.7975284781536 (integer)
        93.82345570870827 (integer)
 
Measurement: vnestStat_performance_transactions_from_type
Tags: from, host, last, node_id, process, type
Fields: failed_request_counter (integer)
        succeed_request_counter (integer)

Database monitoring_op

Information:
Measurements listed in this section are from default Telegraf plugins. Here, measurement name equals plugin name. Refer to plugin documentation for more information.

For example, documentation for Telegraf plugin "cpu" can be found here.

Measurement: cpu
Tags: cpu, host, node_id, tag
Fields: usage_guest (float)
        usage_guest_nice (float)
        usage_idle (float)
        usage_iowait (float)
        usage_irq (float)
        usage_nice (float)
        usage_softirq (float)
        usage_steal (float)
        usage_system (float)
        usage_user (float)
 
Measurement: disk
Tags: device, fstype, host, mode, node_id, path, tag
Fields: free (integer)
        inodes_free (integer)
        inodes_total (integer)
        inodes_used (integer)
        total (integer)
        used (integer)
        used_percent (float)
 
Measurement: diskio
Tags: ID_PART_ENTRY_UUID, SCSI_IDENT_SERIAL, SCSI_MODEL, SCSI_REVISION, SCSI_VENDOR, host, name, node_id, tag
Fields: io_time (integer)
        iops_in_progress (integer)
        read_bytes (integer)
        read_time (integer)
        reads (integer)
        weighted_io_time (integer)
        write_bytes (integer)
        write_time (integer)
        writes (integer)
 
Measurement: linux_sysctl_fs
Tags: host, node_id, tag
Fields: aio-max-nr (integer)
        aio-nr (integer)
        dentry-age-limit (integer)
        dentry-nr (integer)
        dentry-unused-nr (integer)
        dentry-want-pages (integer)
        file-max (integer)
        file-nr (integer)
        inode-free-nr (integer)
        inode-nr (integer)
        inode-preshrink-nr (integer)
 
Measurement: mem
Tags: host, node_id, tag
Fields: active (integer)
        available (integer)
        available_percent (float)
        buffered (integer)
        cached (integer)
        commit_limit (integer)
        committed_as (integer)
        dirty (integer)
        free (integer)
        high_free (integer)
        high_total (integer)
        huge_page_size (integer)
        huge_pages_free (integer)
        huge_pages_total (integer)
        inactive (integer)
        low_free (integer)
        low_total (integer)
        mapped (integer)
        page_tables (integer)
        shared (integer)
        slab (integer)
        swap_cached (integer)
        swap_free (integer)
        swap_total (integer)
        total (integer)
        used (integer)
        used_percent (float)
        vmalloc_chunk (integer)
        vmalloc_total (integer)
        vmalloc_used (integer)
        wired (integer)
        write_back (integer)
        write_back_tmp (integer)
 
Measurement: net
Tags: host, interface, node_id, tag
Fields: bytes_recv (integer)
        bytes_sent (integer)
        bytes_sum (integer)
        drop_in (integer)
        drop_out (integer)
        err_in (integer)
        err_out (integer)
        packets_recv (integer)
        packets_sent (integer)
        packets_sum (integer)
        speed (integer)
        utilization (integer)
 
Measurement: nstat
Tags: host, name, node_id, tag
Fields: IpExtInOctets (integer)
        IpExtOutOctets (integer)
        TcpInErrs (integer)
        UdpInErrors (integer)
 
 
Measurement: processes
Tags: host, node_id, tag
Fields: blocked (integer)
        dead (integer)
        idle (integer)
        paging (integer)
        running (integer)
        sleeping (integer)
        stopped (integer)
        total (integer)
        total_threads (integer)
        unknown (integer)
        zombies (integer)
 
Measurement: procstat
Tags: host, node_id, process_name, tag, user
Fields: cpu_time (integer)
        cpu_time_guest (float)
        cpu_time_guest_nice (float)
        cpu_time_idle (float)
        cpu_time_iowait (float)
        cpu_time_irq (float)
        cpu_time_nice (float)
        cpu_time_soft_irq (float)
        cpu_time_steal (float)
        cpu_time_stolen (float)
        cpu_time_system (float)
        cpu_time_user (float)
        cpu_usage (float)
        create_time (integer)
        involuntary_context_switches (integer)
        memory_data (integer)
        memory_locked (integer)
        memory_rss (integer)
        memory_stack (integer)
        memory_swap (integer)
        memory_vms (integer)
        nice_priority (integer)
        num_fds (integer)
        num_threads (integer)
        pid (integer)
        read_bytes (integer)
        read_count (integer)
        realtime_priority (integer)
        rlimit_cpu_time_hard (integer)
        rlimit_cpu_time_soft (integer)
        rlimit_file_locks_hard (integer)
        rlimit_file_locks_soft (integer)
        rlimit_memory_data_hard (integer)
        rlimit_memory_data_soft (integer)
        rlimit_memory_locked_hard (integer)
        rlimit_memory_locked_soft (integer)
        rlimit_memory_rss_hard (integer)
        rlimit_memory_rss_soft (integer)
        rlimit_memory_stack_hard (integer)
        rlimit_memory_stack_soft (integer)
        rlimit_memory_vms_hard (integer)
        rlimit_memory_vms_soft (integer)
        rlimit_nice_priority_hard (integer)
        rlimit_nice_priority_soft (integer)
        rlimit_num_fds_hard (integer)
        rlimit_num_fds_soft (integer)
        rlimit_realtime_priority_hard (integer)
        rlimit_realtime_priority_soft (integer)
        rlimit_signals_pending_hard (integer)
        rlimit_signals_pending_soft (integer)
        signals_pending (integer)
        voluntary_context_switches (integer)
        write_bytes (integer)
        write_count (integer)
 
Measurement: swap
Tags: host, node_id, tag
Fields: free (integer)
        in (integer)
        out (integer)
        total (integer)
        used (integer)
        used_percent (float)
 
Measurement: system
Tags: host, node_id, tag
Fields: load1 (float)
        load15 (float)
        load5 (float)
        n_cpus (integer)
        n_users (integer)
        uptime (integer)
        uptime_format (string)

Measurement: dtquery_dt_dist_dt_node_id_type
Tags: dt_node_id, process, tag, type
Fields: count_i (integer)
 
Measurement: dtquery_dt_dist_host_dt_node_id
Tags: dt_node_id, process, tag
Fields: count_i (integer)
 
Measurement: dtquery_dt_dist_type_type
Tags: process, tag, type
Fields: count_i (integer)
 
Measurement: dtquery_dt_status
Tags: process, tag
Fields: total (integer)
        unknown (integer)
        unready (integer)
 
Measurement: dtquery_dt_status_detailed_type
Tags: process, tag, type
Fields: total (integer)
        unknown (integer)
        unready (integer)

Measurement: ecs_fabric_agent_dirstat_size_bytes
Tags: host, node_id, path, tag, url
Fields: gauge (float)

Measurement: sr_JournalParser_GC_RG_DT
Tags: DT, RG, last, process
Fields: majorMinorOfJournalRegion (string)
        pendingChunks (integer)
        timestampOfChunkRegion (string)
        timestampOfJournalParserLastRun (string)
 

Measurement: sr_ObjectGC_CAS_RG
Tags: RG, last, process
Fields: STATUS (string)

Measurement: vnestStat_btree
Tags: cumulative_stats, host, level, node_id, tag
Fields: level_count (float)
        page_count (float)
        size_bytes (float)

Database monitoring_vdc

Metrics in this database are calculated values over whole VDC without reference to particular data node.

Information:

Metrics below are aggregated over data nodes for raw measurements used in Grafana ECS UI.

Measurement: cq_disk_bandwidth
Tags: type_op ('read', 'write')
Fields: consistency_checker (float)
        erasure_encoding (float)
        geo (float)
        hardware_recovery (float)
        total (float)
        user_traffic (float)
        xor (float)
 
Measurement: cq_node_rebalancing_summary
Tags: none
Fields: data_rebalanced (integer)
        pending_rebalance (integer)
 
 
Measurement: cq_process_health
Tags: none
Fields: cpu_used (float)
        mem_used (float)
        mem_used_percent (float)
        nic_bytes (float)
        nic_utilization (float)
 
Measurement: cq_recover_status_summary
Tags: none
Fields: data_recovered (integer)
        data_to_recover (integer)