Start a Conversation

Unsolved

This post is more than 5 years old

3728

February 22nd, 2017 06:00

Scaleio RF cache

Hello, we have test scaleio cluster of latest  Version: 2.0.12000

Cluster have nodes with RFCache enabled.

And dmesg -T show such errors:

[Wed Feb 22 15:41:14 2017] SFC-cache Error:Read from the source (Hnd 10013) failed with error 1.

[Wed Feb 22 15:41:14 2017] Read from the source /dev/sdg failed with error 1.

[Wed Feb 22 15:41:14 2017] SFC-cache Error:Read from the source (Hnd 10012) failed with error 1.

[Wed Feb 22 15:41:14 2017] Read from the source /dev/sdl failed with error 1.

[Wed Feb 22 15:41:14 2017] SFC-cache Error:Read from the source (Hnd 10012) failed with error 1.

[Wed Feb 22 15:41:14 2017] Read from the source /dev/sdl failed with error 1.

[Wed Feb 22 15:41:14 2017] Read from the source /dev/sdc failed with error 1.

[Wed Feb 22 15:41:14 2017] SFC-cache Error:Read from the source (Hnd 10008) failed with error 1.

If I disable RFcache  dmesg output is like this:

[Wed Feb 22 15:42:14 2017]  sdq:

[Wed Feb 22 15:42:14 2017]  sdq:

[Wed Feb 22 15:42:14 2017]  sdq:

[Wed Feb 22 15:42:15 2017]  sdq:

[Wed Feb 22 15:42:15 2017]  sdq:

[Wed Feb 22 15:42:15 2017]  sdq:

[Wed Feb 22 15:42:16 2017]  sdq:

[Wed Feb 22 15:42:16 2017]  sdq:

[Wed Feb 22 15:42:16 2017]  sdq:

[Wed Feb 22 15:42:16 2017]  sdq:

[Wed Feb 22 15:42:16 2017]  sdq:

[Wed Feb 22 15:42:16 2017]  sdq:

Is it normal or some kind of bug ?

Thanks,

Matas

306 Posts

February 23rd, 2017 02:00

Hi Matas,

I haven't seen this error. Could you please open an SR so we can look deeper into it?

Thanks,

Pawel

12 Posts

May 22nd, 2017 08:00

What was found with this error?  I am having the same error messages.  ScaleIO 2.0.12000.122 Ubuntu Linux.

Thanks,

Chad

22 Posts

May 22nd, 2017 12:00

Hello, Chad, I wasn't able to solve this problem yet. If you will be lucky, please let us know.

Thanks, 

Matas

306 Posts

May 24th, 2017 02:00

Hi Chad,

No, I could not find such an error in any of the tickets we worked so far, that's why I suggested to open an SR.

Thank you,

Pawel

12 Posts

May 24th, 2017 08:00

We are evaluating Scaleio so I don't have support yet.  I would have opened a ticket with this one already otherwise.  We have a lot of other gear with you guys but haven't convinced anyone yet to replace one of our aging San's with this yet.

I configured one SSD to cache on each of the SDSs.  Running Ubuntu 16.04 on all nodes with the driver_sync correctly syncing from the EMC repository for Xcache.  The rest of the drives in the server are sata HDD.  I tail the syslog or kern.log and within 10 seconds of enabling Flash cache at the SP level I get flooded with these errors for each of the HDD's.  There is no xcache log directory so I assumed all of it is logged at the SDS Log location.  Looking at trc.0 there is nothing significant after enabling Flash cache.  It seems to add all the devices and I get no error in SDS logs.


Kern and syslog errors per device show...

May 23 09:30:28 hostname kernel: [69151.019224] SFC-cache Error:Read from the source (Hnd 10155) failed with error 1.

May 23 09:30:29 hostname kernel: [69151.021394] Read from the source /dev/sdc failed with error 1.

I had a bunch of extra RAM in these servers so I have switched to using 15GB of RAM cache which gave me a significant boost to read performance in my benchmark tests.  Still curious about using RFcache and its performance compared to RAM cache as Ram I would assume to be faster but the SSD can hold a lot more cache-able data.

306 Posts

May 31st, 2017 01:00

Hi Matas,

From what I remember, you do have a support contract, right? If yes, please open an SR and we'll look into it deeper.

Thx,

Pawel

73 Posts

June 8th, 2017 13:00

Carterbury,

There is an issue with RFCache we are chasing down and hope to have fixed soon. It only affects Linux kernels greater than 4.30, which I assume with Ubuntu you are running. Stay tuned in the next couple of weeks for an update to the 2.0.1.3 code which will include this fix.

Thanks!

Rick

22 Posts

June 8th, 2017 13:00

Hi, our kernel 3.10 also have this issue. We run CentOS flavor.

73 Posts

June 9th, 2017 07:00

Hi Matas,

With the older kernel, it will definitely be a different issue than the one being looked at currently. As there were some significant changes made to the Linux kernel at 3.13 and at 4.30. If you can, open a support case and we'll take a look at the logs to see what is going on.

Thanks,

Rick

12 Posts

June 12th, 2017 16:00

I am using this repository so it appears to be related for me.  It looks like I would need to update the OS though in order to apply the fix?

/Ubuntu/2.0.12000.122/4.4.0-62-generic/

12 Posts

August 3rd, 2017 13:00

RickH,

I updated to the latest Ubuntu and Scaleio but seeing the same behavior.  I assume the fix wasn't included in 2.0.1.3?

root@kal-l-sio-fil03:~# dpkg --list |egrep emc

ii  emc-scaleio-gateway                 2.0-13000.211                                amd64        rpm

ii  emc-scaleio-lia                     2.0-13000.211.Ubuntu.16.04                   amd64        scaleio lia package

ii  emc-scaleio-mdm                     2.0-13000.211.Ubuntu.16.04                   amd64        scaleio meta-data manager package

ii  emc-scaleio-sds                     2.0-13000.211.Ubuntu.16.04                   amd64        scaleio sds package

ii  emc-scaleio-xcache                  2.0-13000.211.Ubuntu.16.04                   amd64        scaleio xcache package

73 Posts

August 4th, 2017 07:00

Correct. The fix will be in the 2.0.1.4 code, which is supposed to be due out in early to mid-September.  What you have above is the shipping 2.0.1.3 code. Hope that helps!

12 Posts

August 7th, 2017 07:00

Thanks Rick, 

I will wait until then.  You had originally thought it would make 2.0.1.3 is why I was hopeful after installing.

25 Posts

November 30th, 2017 11:00

Downloading 2.0.1.4 now and will let you know how it goes once its installed and I try RF Cache again.

No Events found!

Top