Highlighted
Mabro1
Berylium

Ask The Expert: Discussing the challenges of Long Distance Links

The challenges of Long Distance Links with RRR &Jon Klaus

Welcome to the EMC Support Community Ask the Expert conversation. This is an opportunity to discuss and learn about the variety of challenges when connecting storage and host systems over long distances.

This discussion begins on Monday, November 19th. Get ready by bookmarking this page or signing up for email notifications.

Your hosts:

profile-image-display.jspa?imageID=7024&size=350

Rob Koper is working in the IT industry since 1994 and since 2004 working for Open Line Consultancy. He started with Clariion CX300 and DMX-2 and worked with all newer arrays ever since, up to current technologies like VNX 5700 and the larger DMX-4 and VMAX 20k systems. He's mainly involved in managing and migrating data to storage arrays over large Cisco and Brocade SANs that span multiple sites widely spread through the Netherlands. Since 2007 he's an active member on ECN and the Support Forums and he currently holds Proven Professional certifications like Implementation Engineer for VNX, Clariion (expert) and Symmetrix as well as Technology Architect for VNX, Clariion and Symmetrix.

https://community.emc.com/profile-image-display.jspa?imageID=6000&size=350

Jon Klaus has been working at Open Line since 2008 as a project consultant on various storage and server virtualization projects. To prepare for these projects, an intensive one year barrage of courses on CLARiiON and Celerra has yielded him the EMCTAe and EMCIEe certifications on CLARiiON and EMCIE + EMCTA status on VNX and Celerra.

Currently Jon is contracted by a large multinational and part of a team that is responsible for running and maintaining several (EMC) storage and backup systems throughout Europe. Amongst his day-to-day activities are: performance troubleshooting, storage migrations and designing a new architecture for the Europe storage and backup environment.

The Event takes place between the 19th - 30th November 2012. Rob and Jon will take your questions during this time.

Also watch out for the tweet chat on #EMCATE about this very same topic taking place on the 26th November.

36 Replies
Mabro1
Berylium

Re: Ask The Expert: Discussing the challenges of Long Distance Links

This discussion is now open for questions to Rob & Jon. We are looking forward to fun interactive and interesting discussion.

Cheers,

Mark

0 Kudos
RRR2
Osmium

Re: Ask The Expert: Discussing the challenges of Long Distance Links

Thanks Mark. I'll inform Jon so he can post the discussiuon starter

0 Kudos
JonK1
Silver

Re: Ask The Expert: Discussing the challenges of Long Distance Links

And we’re off! Welcome to the Ask the Expert session on long distance links. This is the second Ask the Expert session for RRR and me; if you want to read back on CLARiiON/VNX performance, check out this thread. Also, since long distance links are usually installed for replication purposes, be sure to check out Rupal's thread on VNX Replication.

So first of all, a bit of extra background info on RRR and me. We both work for the same EMC partner (Open Line) in the southern part of the Netherlands. 99% of my time is spent with the CLARiiON/VNX range, whereas RRR also occasionally ventures into the Symmetrix range. Both of us run projects for Open Line or are contracted by other companies that need someone to talk storage with them.
I am currently working for a large multinational that has datacenters spread across Europe and the rest of the world. Some are close together, other are thousands of kilometers apart.

RRR has done a lot of storage replication for a different multinational, mostly with Symmetrix systems and has been spending quite some time back at Open Line managing the various mid-range systems, all of them replicating.

A good thing to know about the Netherlands is that it’s a small country. Put your cruise control on the legal speed limit of 130km/u and you’ll drive from the south into the North Sea in a bit more than 2 hours. These relatively short links plus the abundance of dark fibers in the ground mean that we can cross the country with high speed, low latency links. Often it’s as easy as running FC over a dark fiber and we’re done.

This is of course not as simple for other, much larger countries. Let it be distance related, or perhaps a lack of fiber or stable links. RRR and I would like to make this session as interactive as possible. So if you are based in India, the US, the Middle East... any other country that does things differently: let us know! Post about your experiences with long distance links, let us know when you make the switch from FC to IP or what you find easy/difficult about long distance links. This Ask the Expert session will not be simplex… let’s make it duplex!

So what will RRR and I talk about? We’ll briefly touch on the different kinds of replication and what long distance links have to do with this.  We’ll talk about some technologies: long-wave versus short-wave. Playing with colors: CWDM, DWDM & EWDM. Paramount with all long distance links is attenuation: we’ll discuss what it is and how you can manipulate it. We’ll keep talking some more about Fibre Channel: what are buffer credits, why do you need them and how can you make sure you’ve got enough of them. And many more things! If you have specific questions, feel free to throw them out here for us or someone else to answer them.

We hope for a lively ATE session. And please, don’t forget about the #EMCATE tweetchat on the 22nd of November. Check out the details over here.

JonK1
Silver

Re: Ask The Expert: Discussing the challenges of Long Distance Links

Kicking off with a bit of techno: let’s talk attenuation and modal dispersion! Every link you have will suffers from it. What is it?

If you’ve worked with fibers before, you know there’s two types of them: single mode and multimode fiber. The main difference between these fibers is the diameter of the core: single mode fiber is usually 9 microns thick, whereas multimode fiber is either 50 or 62,5 microns thick. (The sign for micron is called mu; guess where RRR's twitter handle of @50mu comes from! )

If you send light through a medium (in our case, fiber) it will decrease in intensity as the distance becomes longer. If you make the link long enough there won’t be any signal left on the other end. This is attenuation, your main distance limitation. One way of solving this is sending more light into the fiber (or putting a more sensitive receiver on the other end). Another way is using fibers that have less attenuation. A real world example is fog while driving your car: either you use more intense headlights and burn through the fog (better interface), or you remove the fog (better quality fiber). Attenuation is measured in dB/km, for example 3dB/km. Very important: each connector also diminishes the signal (0,5-0,75dB is a good guideline), so don’t put too many patch panels in your paths.

Looking at multi-mode fiber, this thick core allows for relatively inexpensive light sources such as LED or multi-mode lasers. The downside of this is that light can enter the fiber at different angles (or modes, hence the name multimode).  Some of the light goes straight through to the end (taking the shortest route) and arrives first at the receiver. Some light enters the fiber at a slightly off-center angle and reflects off the edges of the fiber/cladding. Since bouncing around causes it to take a slightly longer route, this light will arrive slightly later at the receiver. What you end up with is a spread-out signal pulse (less intense and of longer duration). This is modal dispersion, your main speed limitation. If your pulse takes longer to build and diminish, you can’t send as many pulses per second…

A real world example here would be standing in a very long, narrow hallway with a couple of random people, closing your eyes and all running to the end of it. Everyone will take a slightly different path. Mandatory disclaimer here: EMC², RRR or me are NOT responsible for any adverse health effects from this little test!

Of course not all fiber is created equal. Better quality cores in fibers have diminished the attenuation, allowing light to travel further without diminishing in strength as much. To tackle modal dispersion, graded-index multimode fibers have been invented that bend the light back towards the center of the core instead of letting it bounce around.

Multimode fibers are mainly used in short distance links, with cables classified as OM1 through OM4, where OM4 is currently the newest and highest quality 50micron cable. If you look at some of the information Cisco provides on link distances, you can see that the difference is enormous: where OM1 fiber can cover 21m on 8Gbit FC, OM4 fiber does 190m. But if you need to cover more than roughly 190m at high speeds, multi-mode isn’t going to cut it. You need to go single mode…

Single mode uses a more expensive laser that outputs light in a single angle (single mode). There is still some modal dispersion, but it’s much, much less. A single mode laser also outputs more light and can usually work with less light on the receiving end. These differences allow for longer links at higher speeds: an extended reach single mode SFP (wavelength 1550nm) can cover 40km on 9micron single-mode fiber at 8Gbit speeds. Now we’re talking!

So we’ve touched on attenuation and modal dispersion and on how to reduce it. But what if you have a link that’s too long for multi-mode, but too short for true single-mode. For example, a link that’s 1 or 2 kilometers long? If you install multi-mode equipment, there’s no light on the other end. If you install signal-mode transmitters, you run the serious risk of overloading the receiver on the other end: it will not be able to distinguish between the pulses and the pauses between pulses. The end-result is errors.

Each transmitter has an optical sensitivity and a specific output power, measured in dBm. Let’s take for an example a transmitter that has a transmit power of 4dBm and which wants to receive signals in the -6/-12dBm range. At the very least I’ll need to have a link that attenuates 10 dB, preferably even a bit more to be on the safe side, say 12dB. 12dB attenuation would leave you with -8dBm signal strength at the receiver end; not too much, not too little. You don’t want to get too close to the minimal receive value: the transmitting SFP will wear down a bit over the years, so your signal and link need some reserve budget.

If that link is only 5km long @ 0,35dB/km, that will only give me roughly 4,7dB of attenuation. Add four connectors at 0,75dB each and the total link will have an attenuation of 7,7dB. Not enough; I’ll overload the receiver. The solution is adding attenuators: little sun-glasses you plug on your cable or switch port! You can get them in many different flavors: -1dB, -5dB, -10dB… In this case you might choose the -5dB attenuators: your total line attenuation will now be 7,7+5= 12,7dB. This means that the light you send out at 4dBm will end up at the other end with -8,7dBm: smack in the middle of the receiver range. Job done, link operational!

Message was edited by: Jon Klaus Correcting some dBm/dB errors!

Allen_Ward
Ruthenium

Re: Ask The Expert: Discussing the challenges of Long Distance Links

Good morning gentlemen :-)

Well it's morning here at least...   So, I've got a question for you, but I'll start with a little bit of context regarding our environment just so you know where I'm coming from.

I'm in Canada, which is one of those "slightly larger" countries you were referring to. We only really have two dedicated storage replication links that would qualify as "long distance" in our environment. One of them on a scale within a province (similar to your "within a country" and one that reaches halfway across Canada. In our case both links are actually managed by our Network Support group and are SONET links with dedicated FC bridge equipment at either end. The FC "bridges" handle all issues of converting our native FC connections and transferring them over distance (including all buffer credit related issues) so that from my perspective my FC switches at each end think they have a direct connection to each other (with a fair bit of latency, but that can't be helped). So a lot of the issues of long distance links don't affect me directly. I am definitely interested in what is going on at the back end of this though.

Now, to my first question. Why do you refer to signal strength in terms of attenuation instead of absolute values? In your example above it's a bit difficult to grasp why you are saying you want to receive the signal in the -6/-12 dB range. Why do you refer to negative dB? at the receive end and positive dB on the source end?

JonK1
Silver

Re: Ask The Expert: Discussing the challenges of Long Distance Links

Hey Allen, long time no see!

I have to admit I'm always struggling with signal strength myself. It appears I’ve made a mistake in the original post (mixing up dB and dBm). Let me clarify over here and then edit the original post

A transceiver is measured in dBm, which is power in reference to 1 mW (and indeed an absolute value). Attenuation is measured in dB, which is the factor in which a signal is boosted or attenuated.

A transceiver rated at 0 dBM sends out light at 1mW. 3dBm extra doubles the power, so a 3dBm transceiver sends out 2mW of light. Comparably, -3dBM equals 0,5mW.

If we use this info on the example of a transceiver that sends at 4dBm, it will output light at roughly 2,5mW. The receiving end of the transceiver is rated for -6/-12dBm, which means it can work with as little light as 0,25 to 0,06mW and still make bits and bytes of it.

I hope this makes sense.

RRR2
Osmium

Re: Ask The Expert: Discussing the challenges of Long Distance Links

Very nicely explained, Jon!

So every 3dB means double the value. So a laser with an Tx of 9dBm is 3 times double the value, so that would be 1 mW x 2³ = 8 mW. And if the Rx side can receive -12dBm that's 1/16 mW, so the sending power can be 1/128th of what comes out of the Tx?

0 Kudos
JonK1
Silver

Re: Ask The Expert: Discussing the challenges of Long Distance Links

Yep, that's the basic idea of it.

What you see is that the multimode transceivers and the shorter range single mode transceivers usually have ranges that are overlapping, i.e. they would work perfectly fine with a 2m cable attached. As soon as you end up with the longer range single-mode transceivers (especially CWDM/DWDM), you will need a minimal amount of attenuation. How much depends on the specific transceiver.

Allen_Ward
Ruthenium

Re: Ask The Expert: Discussing the challenges of Long Distance Links

The explanation of how they do it makes sense. Thanks guys.

That just leaves the question (which I don't mean you have to answer) of WHY they do it. Seems to me it would make more sense to just talk in terms of actual mW of power... but then it might make it so easy that they don't need the smart guys to work on it :-)

0 Kudos