PowerScale: CloudPools Jobs and Job Engine Jobs

When it comes to CloudPools, there are two distinct types of jobs. There are job engine jobs and there are cloud jobs.
 
Job engine jobs are used only to identify files for archiving. They do not move data to the cloud. Instead, they iterate through the file system identifying files that match filepool policies. Identified files are added to a queue and processed by a cloud job. As each relevant job engine job kicks off, it creates a corresponding cloud job to track the files in the queue. The cloud job is independent of the job engine job; that is, canceling or pausing the job engine job do not affect cloud jobs. Relevant job engine jobs are:

• SmartPools
• SmartPoolsTree
• File Policy
• CloudPoolsLin
• CloudPoolsTreewalk 
  •  The last two are identified as hidden jobs and only appear in certain UI contexts

Cloud jobs are the second stage of the archive process. Once files have been identified and queued for archive by a job engine job, they can be moved to the cloud by a cloud job. The cloud job runs in the background and iterates through the queue of files, writing each out to the cloud. If a cloud job is canceled, it also cancels the corresponding job engine job, if still running.
Cloud jobs are only marked successful or complete if,

• All queued files have finished being transferred to the cloud and
• The associated job engine job has finished identifying files for archive.

Cloud jobs can be started, monitored, canceled, or paused through the CLI only using these commands:

 # isi cloud jobs list   # Show all jobs
 # isi cloud jobs view XX # View details for job XX
 # isi cloud jobs cancel XX # Cancel job XX
 # isi cloud jobs pause XX # Pause job XX
 # isi cloud jobs resume XX # Un-pause job XX
 # isi cloud jobs pause OP_XX #  Pause all jobs of a given operation type (for example: 'isi cloud jobs pause archive  will pause all active archive jobs)
 # isi cloud jobs resume OP_XX # Un-pause jobs of given operation type
 # isi cloud jobs files list XX # View a list of all files associated with job XX
 # isi cloud archive <files or directories> # Start a new archive job   also launches a corresponding job engine job
 # isi cloud recall <files or directories> # Recall previously archived files   also launches a corresponding job engine job

 
It is normal for a cloud job to live on after the job engine job has completed or been canceled. Files are still queued for archive along with a reference to the policy. To stop those archives from happening, issue an isi cloud jobs cancel XX command.
 
Failures for Cloud jobs can be viewed in /var/log/messages, /var/log/isi_cpool_d.log, and /var/log/isi_cpool_io_d.log.