Unsolved
This post is more than 5 years old
6 Posts
0
1367
ScaleIO 2.0.1 split-brain when mounting same volume to several SDCs
Hello,
I installed ScaleIO 2.0.1 system in following variant:
3 servers (CentOS 7):
1. MDM + SDS (2TB HDD) + SDC
2. MDM + SDS (1TB HDD) + SDC
3. TB + SDS (2TB HDD)
(all components stays in same server because this is test installation)
Installation completed via IM with csv-file.
I created protection domain, fault sets, volume. Volume is mapped to SDC on first MDM and to SDC on second MDM. The appeared block device (/dev/scinia) has formatted and mounted to /media/storage/ directory in both servers with SDC.
The problem that is:
I created some file in directory with mounted storage device on server 1. Then I following to server 2 to check that file is appears. But this directory is empty! Then I unmount storage device and mount it again - now i see created file. I tried different filesystems (ext4, ext2, xfs), mapping volume with or without --allow_multi_map option, mounting with different options (_netdev, sync, dirsync) but this did not gave a result. I checked kernel module options but didn't see something to change this behavior.
How to mount storage to several systems and get actual state of content (not split-brain-like situation), like it works when connected to several ESXi hosts?
RHasleton1
73 Posts
1
October 26th, 2017 07:00
Hi Sam,
This has nothing to do with the block storage underneath. The same would happen with any other storage. It has everything to do with the filesystem you are using on both hosts.
For any host to be able to read/write to a volume that is mounted on more than one host AND have the other hosts read/write to the same files/directories (without a dismount), you need a cluster filesystem. ESXi uses VMFS, which is a clustered filesystem. Each host communicates to the other thru various means to let them know they have a lock/reservation and to not touch that particular file when it's open, while sharing the metadata as well to each host.
For Linux, you can try ocfs2, gfs2, GlusterFS, etc. There are a few to choose from. Good luck
Rick
glhf
6 Posts
0
November 2nd, 2017 04:00
Thank you for your answer!
But I dont understand... I mapped same volume to two SDC. If I read binary data from block device (/dev/scinia) on both SDC, and change it (with dd), I see changes only on SDC where I did that. Reading data from scinia on other SDC returns old data. Results must be same, isn't?
Similar effect I see when using clustered filesystems...
RHasleton1
73 Posts
0
November 2nd, 2017 07:00
Sam,
What are describing is exactly the symptoms you would see if you are not using a clustered filesystem. A host does not constantly scan a device for changes made at the block level. That is what the filesystem is for, to be aware of changes made and let the host know. If host A creates a file, how is host B made aware that this file now exists, except that the filesystem is made aware. The only time a rescan happens is on boot or remount of the disk itself, which is why you see if after a remount.
Doing what you are doing now is a good way to corrupt the data on the disk, as the hosts sharing the disk are not aware of each other.
Here are a couple links I found with a quick search:
storage area network - Why are two servers viewing the same drive unable to view the same files? - Server Fault
https://en.wikipedia.org/wiki/Clustered_file_system
If you are seeing the same effect when using a clustered filesystem, I would venture to guess that something is not configured properly on the cluster or the filesystem.
Good luck,
Rick