Cisco:Ports go into Error disabled state immediately

multiple port flaps, which causes TCAM memory usage in certain cases. 

Ports transition immediately to Error disabled state when performing "no shut".

In case of device login, the port goes into error disabled state.
edge-(config-if)# no shutdown
fc4/29: (error) Failed to Write to TCAM

ACL programming fails leading to zone activation failure or device login failure.
Following syslog messages will be seen:

%ZONE-2-ZS_TCAM_SWITCHED_TO_SOFT_ZONING_MODE: %$VSAN 12%$ Switched to soft zoning : Reason: Hard zoning disabled

Reviewed the TCAM entries - 

show system internal acl tcam-usage

TCAM Entries:
=============
                  Region1   Region2    Region3     Region4   Region5   Region6
Mod Fwd   Dir     TOP SYS  SECURITY    ZONING      BOTTOM    FCC DIS   FCC ENA
    Eng          Use/Total Use/Total  Use/Total   Use/Total Use/Total Use/Total
--- ---  ------ ---------- --------- ------------ --------- --------- ---------
4   3    INPUT    239/19664    0/9840 35004/49136* 87/19664    0/0       0/0  

Then this error:

show port internal event-history errors

12) Event:E_DEBUG, length:119, at 424487 usecs after Wed Oct  5 00:50:56 2016
    [102] pi_fsm_port_attr_change_init: Ifindex (fc4/26)0x1199000, Err disabled event (PI_FSM_EV_ACL_MODE_PROG_FAILURE)0x23

Same error exist for:
    [102] pi_fsm_ac_cfg_wait_resp_rcvd: Ifindex (fc4/32)0x119f000, Err disabled (0x404e0005) event (PI_FSM_EV_ACL_MODE_PROG_FAILURE)0x23 due to acl mode failure
  
    [102] pi_fsm_ac_cfg_wait_resp_rcvd: Ifindex (fc4/27)0x119a000, Err disabled (0x404e0005) event (PI_FSM_EV_ACL_MODE_PROG_FAILURE)0x23 due to acl mode failure
   
    [102] pi_fsm_ac_cfg_wait_resp_rcvd: Ifindex (fc4/29)0x119c000, Err disabled (0x404e0005) event (PI_FSM_EV_ACL_MODE_PROG_FAILURE)0x23 due to acl mode failure


AND - 

show port internal event-history errors

12) Event:E_DEBUG, length:119, at 424487 usecs after Wed Oct  5 00:50:56 2016
    [102] pi_fsm_port_attr_change_init: Ifindex (fc4/26)0x1199000, Err disabled event (PI_FSM_EV_ACL_MODE_PROG_FAILURE)0x23

Same error exist for:
    [102] pi_fsm_ac_cfg_wait_resp_rcvd: Ifindex (fc4/32)0x119f000, Err disabled (0x404e0005) event (PI_FSM_EV_ACL_MODE_PROG_FAILURE)0x23 due to acl mode failure
  
    [102] pi_fsm_ac_cfg_wait_resp_rcvd: Ifindex (fc4/27)0x119a000, Err disabled (0x404e0005) event (PI_FSM_EV_ACL_MODE_PROG_FAILURE)0x23 due to acl mode failure
   
    [102] pi_fsm_ac_cfg_wait_resp_rcvd: Ifindex (fc4/29)0x119c000, Err disabled (0x404e0005) event (PI_FSM_EV_ACL_MODE_PROG_FAILURE)0x23 due to acl mode failure

Cisco bug CSCuw15428 - FC Scale : Port error disabled due to tcam rewrite fail

The acl programming fails as the tcam usage has almost reached the limit and complains of no space.

'show system internal acl tcam-usage'

1 1 INPUT 2506/13114 0/3290 65496/68786* 16/13114 0/0 0/0 
1 1 OUTPUT 7/4077 0/1645 0/11469 0/4077 6/1651 9/1654

The high value is due to a number of duplicate entries programmed.

module-1# sh process acltcam fwd-engine 1 input match interface fc 1/15 | inc d80601
00442 44 0 136 d80601 d80601 0 12 - - - - - - - - - | 0 0 0 0 0 0
02802 44 0 136 d80601 d80601 0 12 - - - - - - - - - | 0 0 0 0 0 0

08282 bb 0 136 d80601 6e028d 0 12 - - - - - - - - - | 0 0 0 0 1 0
0854a bb 0 136 d80601 6e028d 0 12 - - - - - - - - - | 0 0 0 0 1 0
08562 bb 0 136 d80601 6e028b 0 12 - - - - - - - - - | 0 0 0 0 1 0
08563 bb 0 136 d80601 6e028b 0 12 - - - - - - - - - | 0 0 0 0 1 0

In some cases, the tcam-usage value goes negative.

`show system internal acl tcam-usage`
TCAM Entries: 
=============
Region1 Region2 Region3 Region4 Region5 Region6 
Mod Fwd Dir TOP SYS SECURITY ZONING BOTTOM FCC DIS FCC ENA 
Eng Use/Total Use/Total Use/Total Use/Total Use/Total Use/Total
--- --- ------ ---------- --------- ------------ --------- --------- ---------
3 3 INPUT 192/9824 0/0 -1126/-1126* 131/9824 0/0 0/0

From the 7.3.0 release onward, instead of a negative value, an unexpectedly large value will be displayed.

1 5 INPUT 598/9824 0/0 4294964129/4294964129* 523/9824 0/0 0/0

Fix:
upgrade to NX-OS 6.2(17).

Workaround:
Reload line card in question

In the instance identified the switch was a 9710 @ 6.2.11c

and the line card in question was a DS-X9448-768K9 (2/4/8/10/16 Gbps Advanced FC Module)