SourceOne Email Management - Index sets marked with "Unperformed Transaction" or "Missing Items" status due to transaction file with .xvlts extension stuck in index DropDir folder and multiple copies of the files found in Intermediary folder

You find one or more SourceOne index transaction files with XVLTS extension are stuck in a processing loop within index DropDir folder and multiple copies of same files would be created within "Intermediary" sub-folder.

If issue is not identified in time it may have following impact:

• Thousands of copies of the XVLTS file with same size and belonging to same index set would end up in intermediary folder. Here is an example of how those moved files within Intermediary may look:

• If multiple transaction files have issue they may end up causing backlog of index transaction files within DropDir folders because maximum number of index processes that could run in SourceOne environment are busy processing files with the issue.

Event messages similar to the following may be found within the ExAsIdxObj.exe.log:

Unable to remove '\\HostName\ES1_MsgCenter\Unpack_Area\Es1Mixed\201710\20180514030827\5B8F6E32246377675963F2E4B99AFF166449CD5FC4E695D200.EMCMF.MD'. OsError: 67|IndexRun.cpp(2062)|Job Id: -1; Activity Name: HostName; Activity Id: -1; Activity Type: -1; HostName (0x86042B76) Unknown error (0x80030043)|IndexThread.cpp(3038)|Job Id: -1; Activity Name: HostName; Activity Id: -1; Activity Type: -1; HostName [\\HostName\Es1_Index\Es1Mixed\201710\001] Aborting index run!!!!!|IndexRun.cpp(1211)|Job Id: -1; Activity Name: HostName; Activity Id: -1; Activity Type: -1; HostName StopAncillaryRun \\HostName\Es1_Index\Es1Mixed\201710\001|IdxAncillaryDB.cpp(295)|Job Id: -1; Activity Name: HostName; Activity Id: -1; Activity Type: -1; HostName Marking local idx state as missmsg E:\ExIndexTemp\Es1Mixed_201710_001\Index|CIdxState.cpp(279)|Job Id: -1; Activity Name: HostName; Activity Id: -1; Activity Type: -1; HostName Es1Mixed_201710_001] Not copying index to network due to previous fatal error. (0x86042B86)|IndexThread.cpp(3279)|Job Id: -1; Activity Name: HostName; Activity Id: -1; Activity Type: -1; HostName

This problem is caused by a software defect.

There are two conditions identified which may cause this issue:

1. Index transaction files have transactions that reference EMCMF files location within Unpack_Area that is inaccessible or may have been changed (in case message center path was changed).
2. Index transaction file contains transaction entries where path to EMCMF files is corrupt or the XVLTS file has been corrupted due to environment issue.

This issue will be resolved in patches or Service Pack versions of EMC SourceOne Email Management post 7.2.SP6 Hotfix 2 (7.2.6.6175). DELL EMC SourceOne Patch and Service Pack kits are available for download via http://support.emc.com. 

Workaround:

1. Stop  EMC SourceOne Index  service on all SourceOne native archive servers with index role. 
2. Check task manager on index server hosts and make sure there is no  ExAsIdxObj.exe  or "ExAsElasticIdxObj.exe" processes are running. SourceOne index service from step 1 will wait for the ExAsIdxObj.exe or ExAsElasticIdxObj.exe to stop before service stops. Executable for index service is ExAsIndex.exe.
3. Once index service from step 1 stops, navigate to index file share and go to \DropDir\Intermediary folder.
4. Based on the files listing within intermediary folder make list index sets impacted. For an example, based on screenshot provided above index set transaction files in question belong to "es1mixed_201710_001" and "es1mixed_201804_001" index sets.
5. Create a folder within Intermediary folder that you will use in the next step to take backup of files from index DropDir folder.
6. Using the list created in step 4 above, identity any files starting with same names, YYYYMM, index set number and move those files to backup folder created in step 5.
7. Start SourceOne index service on all indexing hosts where service was stopped in step 1.
8. Index sets identified above needs to be rebuilt since some of its transaction files didn't get processed. SourceOne Administration Console can be used to submit index sets to be rebuilt. For detailed instruction on how to rebuild index sets SourceOne Email Management Administration Guide can be followed.
9. On successful rebuild of index sets, status of index sets should change to "Available" state from "Unperformed Transaction" or "Missing Items" state.
10. If index sets rebuild successfully, files located within intermediary folder and backup folder (from step 5) related to index sets in question can be deleted.