Isilon: How to enable FTP Tail Mode on OneFS

What is Tail Mode?

Under normal circumstances, when an FTP client downloads a file from the OneFS FTP server, the FTP server opens the file, reads the contents, and closes the connection when it reaches the end of the file. When tail mode is enabled, instead of immediately completing when the FTP server reaches the end of the file in the OneFS file system, it waits for a configurable length of time to see if the file grows (because it is being written to by another process). It continues to make these checks and send the new data to the client until the file no longer increases in size within the timeout interval.

When would you use Tail Mode?

If your workflow transfers large files to the cluster, and also reads data from those files, it is possible to achieve a large improvement in performance and latency if the ingest of the data and reading and streaming happen in parallel rather than waiting for the upload to complete before reading and streaming the file. That is what tail mode enables.

How is Tail Mode enabled?

To enable FTP tail mode on OneFS, it is necessary to edit the /etc/mcp/templates/vsftpd.conf file and add the following entries to the end of the file:

tail_mode_enable=YES
lock_upload_files=NO
The default timeout value is 5 seconds. To change the value to e.g. 10 seconds, add an additional line:

tail_mode_timeout=10

Side Effects of the Setting

All downloads take tail_mode_timeout seconds (default 5) longer to complete due to the timeout waiting for EOF at the end of the transfer.

Upgrade Considerations

Because the changes are made to a template file and not to an override file, the changes are overwritten when the cluster is upgraded to a newer OneFS release. The changes must be manually reapplied to the cluster post upgrade.