UNSOLVED

Matas1

updated

10 years ago

M

Matas1

22 Posts

0

3936

February 22nd, 2017 06:00

Scaleio RF cache

Hello, we have test scaleio cluster of latest  Version: 2.0.12000

Cluster have nodes with RFCache enabled.

And dmesg -T show such errors:

[Wed Feb 22 15:41:14 2017] SFC-cache Error:Read from the source (Hnd 10013) failed with error 1.

[Wed Feb 22 15:41:14 2017] Read from the source /dev/sdg failed with error 1.

[Wed Feb 22 15:41:14 2017] SFC-cache Error:Read from the source (Hnd 10012) failed with error 1.

[Wed Feb 22 15:41:14 2017] Read from the source /dev/sdl failed with error 1.

[Wed Feb 22 15:41:14 2017] SFC-cache Error:Read from the source (Hnd 10012) failed with error 1.

[Wed Feb 22 15:41:14 2017] Read from the source /dev/sdl failed with error 1.

[Wed Feb 22 15:41:14 2017] Read from the source /dev/sdc failed with error 1.

[Wed Feb 22 15:41:14 2017] SFC-cache Error:Read from the source (Hnd 10008) failed with error 1.

If I disable RFcache  dmesg output is like this:

[Wed Feb 22 15:42:14 2017]  sdq:

[Wed Feb 22 15:42:14 2017]  sdq:

[Wed Feb 22 15:42:14 2017]  sdq:

[Wed Feb 22 15:42:15 2017]  sdq:

[Wed Feb 22 15:42:15 2017]  sdq:

[Wed Feb 22 15:42:15 2017]  sdq:

[Wed Feb 22 15:42:16 2017]  sdq:

[Wed Feb 22 15:42:16 2017]  sdq:

[Wed Feb 22 15:42:16 2017]  sdq:

[Wed Feb 22 15:42:16 2017]  sdq:

[Wed Feb 22 15:42:16 2017]  sdq:

[Wed Feb 22 15:42:16 2017]  sdq:

Is it normal or some kind of bug ?

Thanks,

Matas

  • pawelw1

    306 Posts

    1787

    0

    Posted February 23rd, 2017 02:00

    Hi Matas,

    I haven't seen this error. Could you please open an SR so we can look deeper into it?

    Thanks,

    Pawel

  • carterbury

    12 Posts

    1787

    0

    Posted May 22nd, 2017 08:00

    What was found with this error?  I am having the same error messages.  ScaleIO 2.0.12000.122 Ubuntu Linux.

    Thanks,

    Chad

  • Matas1

    22 Posts

    1787

    0

    Posted May 22nd, 2017 12:00

    Hello, Chad, I wasn't able to solve this problem yet. If you will be lucky, please let us know.

    Thanks, 

    Matas

  • pawelw1

    306 Posts

    1787

    0

    Posted May 24th, 2017 02:00

    Hi Chad,

    No, I could not find such an error in any of the tickets we worked so far, that's why I suggested to open an SR.

    Thank you,

    Pawel

  • carterbury

    12 Posts

    1793

    0

    Posted May 24th, 2017 08:00

    We are evaluating Scaleio so I don't have support yet.  I would have opened a ticket with this one already otherwise.  We have a lot of other gear with you guys but haven't convinced anyone yet to replace one of our aging San's with this yet.

    I configured one SSD to cache on each of the SDSs.  Running Ubuntu 16.04 on all nodes with the driver_sync correctly syncing from the EMC repository for Xcache.  The rest of the drives in the server are sata HDD.  I tail the syslog or kern.log and within 10 seconds of enabling Flash cache at the SP level I get flooded with these errors for each of the HDD's.  There is no xcache log directory so I assumed all of it is logged at the SDS Log location.  Looking at trc.0 there is nothing significant after enabling Flash cache.  It seems to add all the devices and I get no error in SDS logs.


    Kern and syslog errors per device show...

    May 23 09:30:28 hostname kernel: [69151.019224] SFC-cache Error:Read from the source (Hnd 10155) failed with error 1.

    May 23 09:30:29 hostname kernel: [69151.021394] Read from the source /dev/sdc failed with error 1.

    I had a bunch of extra RAM in these servers so I have switched to using 15GB of RAM cache which gave me a significant boost to read performance in my benchmark tests.  Still curious about using RFcache and its performance compared to RAM cache as Ram I would assume to be faster but the SSD can hold a lot more cache-able data.

  • pawelw1

    306 Posts

    1793

    0

    Posted May 31st, 2017 01:00

    Hi Matas,

    From what I remember, you do have a support contract, right? If yes, please open an SR and we'll look into it deeper.

    Thx,

    Pawel

  • RHasleton1

    73 Posts

    1793

    0

    Posted June 8th, 2017 13:00

    Carterbury,

    There is an issue with RFCache we are chasing down and hope to have fixed soon. It only affects Linux kernels greater than 4.30, which I assume with Ubuntu you are running. Stay tuned in the next couple of weeks for an update to the 2.0.1.3 code which will include this fix.

    Thanks!

    Rick

  • Matas1

    22 Posts

    1793

    0

    Posted June 8th, 2017 13:00

    Hi, our kernel 3.10 also have this issue. We run CentOS flavor.

  • RHasleton1

    73 Posts

    1793

    0

    Posted June 9th, 2017 07:00

    Hi Matas,

    With the older kernel, it will definitely be a different issue than the one being looked at currently. As there were some significant changes made to the Linux kernel at 3.13 and at 4.30. If you can, open a support case and we'll take a look at the logs to see what is going on.

    Thanks,

    Rick

  • carterbury

    12 Posts

    1084

    0

    Posted June 12th, 2017 16:00

    I am using this repository so it appears to be related for me.  It looks like I would need to update the OS though in order to apply the fix?

    /Ubuntu/2.0.12000.122/4.4.0-62-generic/