Ask The Expert: Discussing the challenges of Long Distance Links

Thanks Mark. I'll inform Jon so he can post the discussiuon starter

And we’re off! Welcome to the Ask the Expert session on long distance links. This is the second Ask the Expert session for RRR and me; if you want to read back on CLARiiON/VNX performance, check out this thread. Also, since long distance links are usually installed for replication purposes, be sure to check out Rupal's thread on VNX Replication.

So first of all, a bit of extra background info on RRR and me. We both work for the same EMC partner (Open Line) in the southern part of the Netherlands. 99% of my time is spent with the CLARiiON/VNX range, whereas RRR also occasionally ventures into the Symmetrix range. Both of us run projects for Open Line or are contracted by other companies that need someone to talk storage with them.
I am currently working for a large multinational that has datacenters spread across Europe and the rest of the world. Some are close together, other are thousands of kilometers apart.

RRR has done a lot of storage replication for a different multinational, mostly with Symmetrix systems and has been spending quite some time back at Open Line managing the various mid-range systems, all of them replicating.

A good thing to know about the Netherlands is that it’s a small country. Put your cruise control on the legal speed limit of 130km/u and you’ll drive from the south into the North Sea in a bit more than 2 hours. These relatively short links plus the abundance of dark fibers in the ground mean that we can cross the country with high speed, low latency links. Often it’s as easy as running FC over a dark fiber and we’re done.

This is of course not as simple for other, much larger countries. Let it be distance related, or perhaps a lack of fiber or stable links. RRR and I would like to make this session as interactive as possible. So if you are based in India, the US, the Middle East... any other country that does things differently: let us know! Post about your experiences with long distance links, let us know when you make the switch from FC to IP or what you find easy/difficult about long distance links. This Ask the Expert session will not be simplex… let’s make it duplex!

So what will RRR and I talk about? We’ll briefly touch on the different kinds of replication and what long distance links have to do with this.  We’ll talk about some technologies: long-wave versus short-wave. Playing with colors: CWDM, DWDM & EWDM. Paramount with all long distance links is attenuation: we’ll discuss what it is and how you can manipulate it. We’ll keep talking some more about Fibre Channel: what are buffer credits, why do you need them and how can you make sure you’ve got enough of them. And many more things! If you have specific questions, feel free to throw them out here for us or someone else to answer them.

We hope for a lively ATE session. And please, don’t forget about the #EMCATE tweetchat on the 22^nd of November. Check out the details over here.

Kicking off with a bit of techno: let’s talk attenuation and modal dispersion! Every link you have will suffers from it. What is it?

If you’ve worked with fibers before, you know there’s two types of them: single mode and multimode fiber. The main difference between these fibers is the diameter of the core: single mode fiber is usually 9 microns thick, whereas multimode fiber is either 50 or 62,5 microns thick. (The sign for micron is called mu; guess where RRR's twitter handle of @50mu comes from! )

If you send light through a medium (in our case, fiber) it will decrease in intensity as the distance becomes longer. If you make the link long enough there won’t be any signal left on the other end. This is attenuation, your main distance limitation. One way of solving this is sending more light into the fiber (or putting a more sensitive receiver on the other end). Another way is using fibers that have less attenuation. A real world example is fog while driving your car: either you use more intense headlights and burn through the fog (better interface), or you remove the fog (better quality fiber). Attenuation is measured in dB/km, for example 3dB/km. Very important: each connector also diminishes the signal (0,5-0,75dB is a good guideline), so don’t put too many patch panels in your paths.

Looking at multi-mode fiber, this thick core allows for relatively inexpensive light sources such as LED or multi-mode lasers. The downside of this is that light can enter the fiber at different angles (or modes, hence the name multimode).  Some of the light goes straight through to the end (taking the shortest route) and arrives first at the receiver. Some light enters the fiber at a slightly off-center angle and reflects off the edges of the fiber/cladding. Since bouncing around causes it to take a slightly longer route, this light will arrive slightly later at the receiver. What you end up with is a spread-out signal pulse (less intense and of longer duration). This is modal dispersion, your main speed limitation. If your pulse takes longer to build and diminish, you can’t send as many pulses per second…

A real world example here would be standing in a very long, narrow hallway with a couple of random people, closing your eyes and all running to the end of it. Everyone will take a slightly different path. Mandatory disclaimer here: EMC², RRR or me are NOT responsible for any adverse health effects from this little test!

Of course not all fiber is created equal. Better quality cores in fibers have diminished the attenuation, allowing light to travel further without diminishing in strength as much. To tackle modal dispersion, graded-index multimode fibers have been invented that bend the light back towards the center of the core instead of letting it bounce around.

Multimode fibers are mainly used in short distance links, with cables classified as OM1 through OM4, where OM4 is currently the newest and highest quality 50micron cable. If you look at some of the information Cisco provides on link distances, you can see that the difference is enormous: where OM1 fiber can cover 21m on 8Gbit FC, OM4 fiber does 190m. But if you need to cover more than roughly 190m at high speeds, multi-mode isn’t going to cut it. You need to go single mode…

Single mode uses a more expensive laser that outputs light in a single angle (single mode). There is still some modal dispersion, but it’s much, much less. A single mode laser also outputs more light and can usually work with less light on the receiving end. These differences allow for longer links at higher speeds: an extended reach single mode SFP (wavelength 1550nm) can cover 40km on 9micron single-mode fiber at 8Gbit speeds. Now we’re talking!

So we’ve touched on attenuation and modal dispersion and on how to reduce it. But what if you have a link that’s too long for multi-mode, but too short for true single-mode. For example, a link that’s 1 or 2 kilometers long? If you install multi-mode equipment, there’s no light on the other end. If you install signal-mode transmitters, you run the serious risk of overloading the receiver on the other end: it will not be able to distinguish between the pulses and the pauses between pulses. The end-result is errors.

Each transmitter has an optical sensitivity and a specific output power, measured in dBm. Let’s take for an example a transmitter that has a transmit power of 4dBm and which wants to receive signals in the -6/-12dBm range. At the very least I’ll need to have a link that attenuates 10 dB, preferably even a bit more to be on the safe side, say 12dB. 12dB attenuation would leave you with -8dBm signal strength at the receiver end; not too much, not too little. You don’t want to get too close to the minimal receive value: the transmitting SFP will wear down a bit over the years, so your signal and link need some reserve budget.

If that link is only 5km long @ 0,35dB/km, that will only give me roughly 4,7dB of attenuation. Add four connectors at 0,75dB each and the total link will have an attenuation of 7,7dB. Not enough; I’ll overload the receiver. The solution is adding attenuators: little sun-glasses you plug on your cable or switch port! You can get them in many different flavors: -1dB, -5dB, -10dB… In this case you might choose the -5dB attenuators: your total line attenuation will now be 7,7+5= 12,7dB. This means that the light you send out at 4dBm will end up at the other end with -8,7dBm: smack in the middle of the receiver range. Job done, link operational!

Message was edited by: Jon Klaus Correcting some dBm/dB errors!

This discussion is now open for questions to Rob & Jon. We are looking forward to fun interactive and interesting discussion.

Cheers,

Mark

Good morning gentlemen :-)

Well it's morning here at least...   So, I've got a question for you, but I'll start with a little bit of context regarding our environment just so you know where I'm coming from.

I'm in Canada, which is one of those "slightly larger" countries you were referring to. We only really have two dedicated storage replication links that would qualify as "long distance" in our environment. One of them on a scale within a province (similar to your "within a country" and one that reaches halfway across Canada. In our case both links are actually managed by our Network Support group and are SONET links with dedicated FC bridge equipment at either end. The FC "bridges" handle all issues of converting our native FC connections and transferring them over distance (including all buffer credit related issues) so that from my perspective my FC switches at each end think they have a direct connection to each other (with a fair bit of latency, but that can't be helped). So a lot of the issues of long distance links don't affect me directly. I am definitely interested in what is going on at the back end of this though.

Now, to my first question. Why do you refer to signal strength in terms of attenuation instead of absolute values? In your example above it's a bit difficult to grasp why you are saying you want to receive the signal in the -6/-12 dB range. Why do you refer to negative dB? at the receive end and positive dB on the source end?

Hey Allen, long time no see!

I have to admit I'm always struggling with signal strength myself. It appears I’ve made a mistake in the original post (mixing up dB and dBm). Let me clarify over here and then edit the original post

A transceiver is measured in dBm, which is power in reference to 1 mW (and indeed an absolute value). Attenuation is measured in dB, which is the factor in which a signal is boosted or attenuated.

A transceiver rated at 0 dBM sends out light at 1mW. 3dBm extra doubles the power, so a 3dBm transceiver sends out 2mW of light. Comparably, -3dBM equals 0,5mW.

If we use this info on the example of a transceiver that sends at 4dBm, it will output light at roughly 2,5mW. The receiving end of the transceiver is rated for -6/-12dBm, which means it can work with as little light as 0,25 to 0,06mW and still make bits and bytes of it.

I hope this makes sense.

Yep, that's the basic idea of it.

What you see is that the multimode transceivers and the shorter range single mode transceivers usually have ranges that are overlapping, i.e. they would work perfectly fine with a 2m cable attached. As soon as you end up with the longer range single-mode transceivers (especially CWDM/DWDM), you will need a minimal amount of attenuation. How much depends on the specific transceiver.

Very nicely explained, Jon!

So every 3dB means double the value. So a laser with an Tx of 9dBm is 3 times double the value, so that would be 1 mW x 2³ = 8 mW. And if the Rx side can receive -12dBm that's 1/16 mW, so the sending power can be 1/128th of what comes out of the Tx?

The explanation of how they do it makes sense. Thanks guys.

That just leaves the question (which I don't mean you have to answer) of WHY they do it. Seems to me it would make more sense to just talk in terms of actual mW of power... but then it might make it so easy that they don't need the smart guys to work on it :-)

There you go! That’s exactly why they talk deciBell jibberish. Even Penny would understand and we can’t have that, can we?

Yesssss. I'll be the slightly normal-ish one and Jon's the weirdo, hahahaha

Or you can be the slightly more normal one and he can be the smarter one :-)   It's all in your perspective!

Agreed! RRR, move over.. you're in my spot!

So this was about getting the link "up" on both ends, but what about actually sending data accross those links? Longer links can actually contain multiple fibre channel frames!! The speed of light isn't infinate, so it takes time (micro seconds to sub milli seconds) for a frame to actually travel from Tx (transmitter) to Rx (receiver).

If a fibre optic path is an FC ISL, then in knowing its line bit-length, we can calculate the maximum number of frames that can simultaneously fit (source to destination and back) on that physical link. That number is the exact anmount of buffer to buffers credits your ISL switch ports will require.

Why?

1. To safely store-and-forward frames at the source end, while they are still in transit to the destination port, and acknowledged as having been received. So the sending switch will keep that frame in a buffer while the ACK hasn't been received.
2. To provide a “successfully sent” I/O acknowledgment to the sending port or HBA so it does not have to wait for the frame to reach the other side of the link, and the acknowledgment that has to come back.

So if you upgrade a port's speed from 2 to 4 Gbps for example, the amount of frames that can be on the link simultaniously doubled and the amount of B2B credits has to be doubled as well.

Please note that although the link between sites can be fast enough and have enough buffer credits to facilitate line rate communication, the receiving array can still be a bottleneck which can cause the sending side to slow down. Consider for example MirrorView or SRDF. The primary storage array is written to by a number of hosts and the the back end of the primary array can handle the amount of IOps. All I/Os that were written are also sent to the secondary array to be replicated. If the receiving end cannot handle that amount of (write) IOps because the number of disks is too low or there's not enough cache available, the receiving (busy) array is processing the incoming I/Os not as fast as the ISL between the sites can provide and so acknowledgements aren't sent back to the primary array instantaniously, but a little bit slower. So in the end the sending application receives this acknowledgement somewhat later and we all know that comminucation is built on send / receive and ACK, right? If the ACK comes in slow, the conversation is slower than you'd like it to be.

So configuring the link between sites is one thing, sizing the recveiving array is another (important) thing!

Before we begin an example of how to calculate buffer requirements, it is important to know the numerical definition of a Fibre Channel Gigabit, as well as to understand the structure of a Fibre Channel Frame.

In the Fibre Channel world, one gigabit is defined to be 1,062,500,000 bits (which is not 1024x1024x1024). Other Fibre Channel Gigabit values are then derived from this reference definition. For example, two (Fibre Channel) Gb = 2 x 1,062,500,000 bits = 2,125,000,000 bits. To avoid confusion with the traditional (non Fibre Channel) definition of a Gb, throughout this document I will use the symbol Gbfc to mean “1,062,500,000 bits”, or 1 Fibre Channel Gb.

In summary:

1 Gbfc = 1,062,500,000

2 Gbfc = 2,125,000,000

4 Gbfc = 4,250,000,000

8 Gbfc = 8,500,000,000

10 Gbfc = 10,625,000,000

16 Gbfc = 17,000,000,000

Next, we show the anatomy of a Fibre Channel Frame with notes.

Start of Frame: 4 bytes or 32 bits

Standard Frame Header: 24 bytes or 192 bits

Data (payload): [0 – 2,112] bytes or [0 – 16,896] bits

CRC: 4 bytes or 32 bits

End of Frame: 4 bytes or 32 bits

TOTAL (Nbr bits/frame): [36 – 2,148] bytes or 288 – 17,184 bits

Notes:

The term byte used here means 8 bits (not the 10 bits that result from 8/10 bit encoding).

The maximum Fibre Channel frame size is 2,148 bytes.

The final frame size must be a multiple of 4 bytes. Thus the Data (payload) segment will, as necessary, be padded with 1 to 3 “fill-bytes” to achieve an overall 4 byte frame alignment.

The standard Frame Header size is 24 bytes. However, up to 64 additional bytes (for a total of an 88 byte header) can be included for applications that need extensive control information. Since the total frame size cannot exceed the maximum of 2,148 bytes, these additional Header bytes will subtract from the Data segment size by as much as 64 bytes (per frame). This is why the maximum Data (payload) size is 2,112 (because [2,112 – 64] = 2,048, which is exactly 2K-bytes of data).

The final frame, once constructed, is passed through the 8 byte to 10 byte conversion process.

In the FC world, 1 Word = 4 x 8/10 bit encoded bytes (40 bits).

Then we have the speed of light:

299,792.458 km / s == 0.00000333564095 s / km, so that's 3.33564095 micro seconds per kilometer, that's 3.3 millionth of a second for each km "traveled".

NOTE: this is in vacuum! I'll adjust the outcome later. The speed of light in fiber is approximately 200,000 km per second instead of almost 300,000.

Next we take the linear distance between the two sites and determine:

1. How many seconds it takes for 1-bit to travel the one-way distance linear between the two sites; that is, express the “distance” between the two sites in seconds. This is determined by the speed of light.
2. Having determined the one-way distance in seconds between the two sites (a fixed number), we can now determine the maximum number of bits that can exist, in transit, between the two sites at any one time. In other words, we can calculate the “distance” between the two sites in bits (as opposed to miles). This is determined by the speed of light, as well as by the rate at which the transmitting equipment (e.g. a fibre channel port) can create electrical variations onto the medium (i.e. fibre optic line). In other words, how fast can it push bits onto one end of the fibre optic line (usually 1 Gbfc, 2 Gbfc, 4 Gbfc or 10 Gbfc).

Let say, for the purposes of discussion, that we have redundant (i.e. two) fibre optic paths between a primary and secondary site.

The linear distance (as opposed to displacement) is 91.732608 Km (or about 57 miles) for one path, and 43.452288 km (or about 27 miles) for the other path.

Suppose the link speed is 2 Gbfc:

Again: this is in vacuum instead of fiber! But stay with me, I'll fix it later on!!

Calculations notes for the tables above:

Column 2 indicates the amount of time (in seconds) it takes one OPTICAL VARIATION (however many number of bits that optical variation happens to represent) to travel the one-way distance specified in column one. It comes from multiplying the number of seconds it takes light to travel 1 km (i.e. 0.00000333564095 s / km), by the number of km traveled which, is specified in the first Column.

Column 3 is the product of column 2 (i.e. the line distance in seconds) and the FC bit insertion rate (2,125,000,000). This column effectively represents the amount of additional I/O that could have been processed at the host, had the response to that I/O been instantaneous; (actually twice that amount, since acknowledgment time for those I/O’s, coming back the other way, have to be accounted for as well).

Column 4 is Column 3 divided by 10. This is done to group single bits in transit into equivalent 8/10 bit “byte” quantities. The Fibre Channel protocol converts every 8 bit byte into a 10 bit equivalent (via the 8/10 bit encoding algorithm) before transmitting it. So the value in this Column 4 will determine the number of buffers needed for an ISL switch port.

Frame Length8 (in km) = (299,792.458 km/s) x (Seconds-Between- Inserted-Bits secs/bit) x (Number-Of-Bits-Per-Frame bits)

Frame Length10 (in km) = (299,792.458 km/s) x (Seconds-Between- Inserted-Bits secs/bit) x (Number-Of-Bits-Per-Frame bits) x 10/8

Where:

Seconds-Between-Inserted-Bits = 1/1,062,500,000 (for 1Gbps FC) = 1/2,125,000,000 (for 2Gbps FC)

Number-Of-Bits-Per-Frame = Variable depending on Data payload and/or Header size. See table and notes above. In most cases the Header will be the Standard size of 24 bytes.

And again: this is vacuum, not in fiber!!! Stay with me a little bit longer....

Notes:

Column 1 represents the total number of 8/10 bits in a Fibre Channel frame (with a standard header size of 24 bytes) at varying Data payloads (PL).

Column 2 represents the product of the 10 bit value in Column 1, and (1/2,125,000,000). Thus, this Column (Column 2) essentially represents the linear distance that 1 (one) single frame consumes for the specified payload (PL).

Column 3 represents the quotient derived by dividing the longest (worst case) DWDM line distance (in kilometers), by the number of kilometers per frame (calculated in Column 2). Thus, this column essentially indicates how many additional ONE-WAY frames worth of data could have been processed by the host/application, had the response to the first frame been instantaneous. In other words, this is how many ONE-WAY (not round trip) switch buffers you would need to allow non-stop transmission. Double (i.e. round trip) the values in this column 3 to yield the number of buffers required of your ISL switch ports.

So in our example for running 2 long links using 2 Gbfc speed (in vacuum):

• the number of B2B credits needed for the 91.7 km leg will be 2 x 30.2 = 60.4, so rounding up to 61 B2B credits.
• the number of B2B credits needed for the 43.4 km leg will be 2 x (43.4/91.7) = 2 x 14.3 = 28.6, so rounding up to 29 B2B credits.

Can't we make a rule of thumb out of this?

I admit, the numbers I used are taken from an example document I have lying around so I didn't need to the math myself , but making a ROT out of it backfires at me.

The easiest way to calculate the rule of thumb is getting back to 1 km, so at 2 Gbfc speed the number of B2B credits needed for a 1 km link is 2 x 30.2170 / 91.732608 = 0.6588 IN VACUUM.

Other examples we can now use to more easily remember these "nice" numbers:

2 Gbfc over a 2 km link = 2 x 0.6588 = ± 1.32

4 Gbfc over a 2 km link = 4 x 0.6588 = ± 2.64

4 Gbfc over a 20 km link = 40 x 0.6588 = ± 26.4

So what would be a nice number to remember? 4 Gbfc requires 1.3 x the number of km? 1.33 x the number of km? I guess we have just created a rule of thumb! 1.33 can be remembered very easily! Right?

So running a 4Gb link over 44 km requires 1.33 x 44 = 58.52 := 59 B2B credits.

A 2 Gbfc link using the same distance (44 km) will need half of that, so 30 B2B credits. So half the speed, half the buffer credits, double the speed, double the buffer credits.

HOWEVER: the speed of light in vacuum needs to be devided by the refraction index of fiber, which is about 1.5.

Buffers Rule of Thumb (taking the actual speed of light into account):
4 Gbfc needs 1.33 times 1.5 times the distance in kilometer = 1.995 buffers per kilometer

This extra 1.5 is because of the refraction index of the carrier used. Most fibers have an index of approximately 1.5, which will "slow down" the speed of light to about 200,000 kilometers per second.

A long distance connection running at 4 Gbfc needs 2 buffers per kilometer

A long distance connection running at 4 Gbfc needs 3.22 buffers per mile

(from km to mile you need to multiply the outcome in km by 1.609, which is not that easy to remember, but you can figure out a number that works for you, right?)

EMC as well as Brocade recommend adding an extra 20% B2B credits to the amount you just calculated.

Message was edited by: RRR Added 20% extra similar to the bestp practices by EMC and Brocade.