Ask the Expert: Customers Save Time with Optic Cable Cleaning

Always good to get different perspectives and experiencess and while I have a lot of sympathy with the "nuclear option" as it were.  Cycling back to the stats the evidence from Bills testing, my testing and industry guys like Fluke.  emonstrates that to replace the SFP as the first diagnostic step has >80% probability it will not fix your issue.  CRC errors are for sure physical layer errors, and they are measured normally at the receiving optic so if I were fault finding CRC errors anywhere in a network the last component to would replace would be the SFP.  But as I said I have the luxury of being detached and scientific, I for sure understand the pressure of keeping a production environment up and running.

Brian,

how do you connect your gear that is spread all over that datacenter floor ?  I hate running 30m home-runs.

Rarely is the optimal word wrt optics.  As explained earlier, given the opportunity to examine SFPs, we have seen the vast majority of the problems NOT in the SFPs, but in the network itself.  This has been supported by industry analysis as well.  An SFP is replaced because it is the first reaction but the problem often resurrects again a short time later.

While the process to replace the SFP is simple in itself, the overall time and expense is often unnecessary and induces delays in problem resolution.  As a first step, cleaning of the cable before insertion is always an easy precaution.  Knowing that you follow cleaning as a best practice helps to narrow problem resolution. 

Hi Dynamox,

It really depends on how large the data center is. Are you running an enterprise class SAN fabric (Principle/Subordinate)?  Like I said, I know that you've got a handle on this and running shorter cables is the right idea but the patch panel thing, meh.  Always been a thorn in my side.

30m cables are a PITA I agree but to be honest, the less number of hops to get from host to storage, the better off you are. If you could run a 30M cable and avoid 1/2/3 patch panels, the less chance you have of port errors.  Just MHO.

FYI I ran several large data centers and ran even LONGER cables to avoid patch panels where needed. Where we did have to use patch panels almost all F/C errors were based on either the patch panel or some sloppy CE who stepped/bent/pulled F/C cable.  Stretching/yanking/pulling on F/C cables is bad mojo!

Brian,

About 8 years ago we started using overhead trays versus under the floor. Under floor trays were just a mess because you had thick power cables run underneath , some people ran cat5 in the same trays.  With overhead trays is much much cleaner. We have 4 port telco racks where MDS 9513s are mounted. Right above them we have distribution patch panels that connect various parts of the datacenter.  In the server area there is a overhead patch panel that serves 3-4 racks. Don't need as many these days as everything is pretty much in UCS chassis.  So my patch cables are 3m long, connectivity looks like this:

MDS -- 3m patch cable --- Patch Panel ------------------ Patch Panel ---- 3m patch cable --- UCS Fiber Interconnect.

Patch Panel to Patch Pannel connectivity is installed by 3rd party contractors who do a lot of cleaning and validating fiber before they declare "ready for production". So we are responsible just for the patch cables.

same here. Cable lengths vary from 2m to 15m. It all depends on where the patchpanels are. And we clean EVERY unplug/plug action.

Thanks for the update, Dynamox.  IMHO, I DESPISE patch panels. The more hops you have between the array and the host the more overhead latency you have.  You have short cables which are great!  I personally would target the patch panels if you're sure the cables haven't been mashed. Like I said, SFPs rarely go bad but they can.

In other words, I know you've got a handle on this and know your environment!  I'll bow out since I really can't add anything what you've already said. I know this is frustrating....

I know 1 Brocade customer where SFP's are swapped quite regularly. Not daily or monthly, but maybe 5 - 10 per year and with 440 switch ports and just as many SFPs on the server-side, I think that's quite often. I also know environments with Cisco equipment with just as many ports where they rarely swap SFPs (1 per year maybe?). How is that even possible? Are Cisco SFPs better?

RRR good observation, not trying to be smart, but without more detail its almost impossible to answer.  I would say from my experience there is no difference in quality in the optics.  I would make two observations one based on this discussion and one on my general experience.

First on one site i.e. the Cisco, they  could be doing as Brian is and cleaning before insertion.  The second is the error handling in the two vendors code.  Not suggesting Brocade call our more events but have more counters that could be interpreted as a call out.  Also some of the event thresholds are configurable so if the thresholds are aggressive you can get the symptoms you describe.

Brian 100%, no IMHO in it, each patch will insert 0.5dB of loss to the overall light budget.  Not drastic but with sererval patches in a run you can end up reducing the signal strength, so that not much contamination is then required to put the link into the marginal range.   really glad to hear you are using the clean before insertion methodology.  That is were you get most return from that effort.

Hi Scott, answering your question from last night about who can clean the optic cables.

The Optic Cable Cleaning procedure is very easy.  Anyone responsible for managing your optic cable plant can perform it.  Pretty much, if I can clean a cable, anyone can! 

What types of network errors might be evidence of a dirty optic cable?

Roberto, again not meaning to be a smart xxx but what network error do you have ;-).  Contamination causes many effects not just ones associated with signal strength line "loss of signal" or "loss of sync".  But can cause events that may not be measured by a counter at the ingress port.

Phenomena like back reflection or optical return loss ORL, it creates a negative dB or if you like “noise” in the fiber.  The general knowledge on OLR is it is the bane of good performance.  Also as the bandwidth increases to 16Gb and 32Gb we see the modulation of the signal getting tighter and tighter, as we cannot make light travel any faster, meaning the effect of the noise can be more profound.

So what does all that techie speak mean.  It means I have seen the removal of contamination actually fix, connection loss issue, solid and flapping, performance issues, both sustained i.e backups and transient.  I have seen it cause loss of connection also at higher levels not just at the fiber connector but affecting host to storage logical connections.  This is why fiber cleanliness is so fundamental to good SAN health, espcially at higher data rates.

We use these for LC and SC SFPs and the dry clean thingy for the patch cables.

Thanks for the reply RRR,

I have also had some good success using the "dry cleaning thingy"   after I've used a wet cleaning solution first.  The few times I've needed to clean an optic, I haven't had much success with the cleaning pens.  I've had better success cleaning optics using a Smart Cleaning Stick as shown below.