Question about ESA-2015-114

Hello Dynamox,

There is an apache server update required to address these CVEs and that was something that required extensive testing on multiple components. The fix will only be released in an MR at this time.

Please let me know if there is anything else I can do for you!

Katie,

do you have a timeline when MR will be released ?  I just upgraded to 7.1.1.4 , turning around and scheduling another OneFS upgrade is very painful. Do developers/support realize that ?

dynamox,

We do understand that being told to upgrade, is inconvenient and impactful.

I just want to give a little more info on why fixes are put into an MR as apposed to a patch.

There are two different areas in OneFS that code changes may be needed to fix any given issue.

Kernel space and user space.

If the fix needs to be addressed in the kernel this is put into an MR.

Most of reason being a reboot would be needed regardless as the kernel would need to be reinitialized to pick up the code changes. So these are rolled up in an MR with other fixes that would need a reboot as well. Hitting multiple issues in one reboot instead of one off patch install/reboot for each individual issue.

If the fix can be addressed in user space a patch can be used however sometimes these also get rolled into an upgrade. These are typically applied on the fly and only require restarting certain processes pertaining to the code that needed to be changed.

The fix to the apache server is kernel related, so based on the above it is put into an MR release.

7.1.1.5 is currently available, Download below:

https://download.emc.com/downloads/DL59656_Isilon_OneFS_7.1.1.5_installation_file.tar.gz

and release notes as well:

https://support.emc.com/docu55585_OneFS-7.1.1-MR-Release-Notes.pdf?language=en_UShttps://download.emc.com/downloads/DL59656_Isilon_OneFS_7.1.1.5_installation_file.tar.gz

Thanks Shane,  are there plans to move Apache to user space ?  I can't schedule OneFS upgrade every time there is a new vulnerability in Apache (seems like every week there is one).

Shane:

since apache is (in parts ) in userspace are there plans to make apache only available for specific accesszones (i.e. System) so other accesszones (userzones) are unable to connect to the GUI? AFAIK this is currently not the case.

Thanks & Regards

sluetze

Not sure for release prior to 7.1.1.x but in 7.1.1.x you can only use the WebUI from the system zone.  If you connect to an IP that's not in the system zone you will get access denied.

Example from my 7.1.1.4 env.  Notice how IP 192.168.33.207 is in a zone called 'zoneA' and the resulting error when connecting to the WebUI management interface on that IP.

sluetze,

Yan is correct, since Access zone have been introduced this has been the case.

However this is only completely possible in 7.2.0.x and up, as NFS was added to likewise and now allows you to have NFS in a different zone other than System.

dynamox,

Apache is in user space, but has dependencies in the kernel. These kernel dependencies needed addressing for the CVE.

So pending where the fix for Apache exists depends on how the fix is addressed; in this case kernel dependencies.

ok, i am curious how other customers are handling what seems to be a never ending list of ESA. Do you actually upgrade OneFS every time ESA comes out (that requires kernel patching). 

I would like to at least "un-bind" Apache from specific interfaces to limit exposure ( public user facing network versus internal management network).

At the end of the day what can Isilon do to help me from running to the business every month and ask them for permission to reboot the cluster ?

Dynamox I am in the same situation.  Unplanned outages are bad, but it is getting to the point where an outage is an outage planned or unplanned and I think in the industry management is just starting to take notice that a product does not really fit the bill of having high uptime if you have to take it down every other month for maintenance.

dynamox,

I unfortunately do not have insight, into how other customers deal with ESA's and upgrades. As I am a support engineer, and deal mostly with break fix.

I would like to make the recommendation to reach out to your EMC account team as they work in a much broader scope in regards to this.

While there is not away to shut down Apache on certain interfaces. You could use access zone to separate WebUI access from public facing and management interfaces. Which is a side discussion going on in this thread now.

T_Koopman wrote: Dynamox I am in the same situation.  Unplanned outages are bad, but it is getting to the point where an outage is an outage planned or unplanned and I think in the industry management is just starting to take notice that a product does not really fit the bill of having high uptime if you have to take it down every other month for maintenance.

yep, Windows guys are begging to make fun of me, saying i reboot my Isilon cluster more often than they reboot their boxes.

https://support.emc.com/kb/16602

Back to using the above KB to limit what IP can connect to Isilon via https.   Limits exposure but does not fix issue.

Shane Dekart wrote: While there is not away to shut down Apache on certain interfaces. You could use access zone to separate WebUI access from public facing and management interfaces. Which is a side discussion going on in this thread now.

but Apache is still listening on port 80 so it's still suseptible to an attack.

shane / Yan,

I did not make myself clear. Okay access is limited from zones outside System - this is a progress. But as dynamox wrote, the service is still listening on this port - so it may be possible for attackers to exploit that.

dynamox : we may have weekly maintenance windows (to patch / boot the isilon), but we are unable to use each one of them to take down our clusters. The customer would not like that - it's more something political / strategic than a question of possibilities. It just does not make a good impression towards the customer if I take the service down, that he pays for. (this does NOT mean, that I let the services unprotected. it does only mean I have to evaluate my possibilitys and may "bundle" some patches to lower the amount of downtimes)

In some situations it could be useful to set up the isilon behind firewalls, and only allow specific ports/protocols (i.e. HTTP or HTTPS) only from specific subnets (admin-server, admin-clients) this can limit the possibilities. but this is not a fix - this is a workaround.