Unsolved
This post is more than 5 years old
9 Posts
0
1276
Container restart kills Storage Pool
Hi,
After an epic struggle, I had a working ECS Single Node. I then (foolishly) restarted the container. Rookie error.
Since the restart, the GUI is showing the node as "Suspect", and the available capacity of the Pool as 0.0GB. I need slightly more storage than that to test my application.
Authentication is working via the API, and I can log into the GUI (obviously). There are a bunch of recurring errors/warnings in the log, but these were showing up prior to the container restart:
2015-11-20 08:59:53,377 [SSTable-000] WARN PartitionBlocksContainer.java (line 189) Could not reallocate large blocks due to limitation. Required size 69658934400, large block limit 0, SS 10.x.x.x, partition 1
2015-11-20 08:59:53,377 [SSTable-000] WARN PartitionSelector.java (line 386) Cannot allocate block from partition 1 for size 69658934400 SS 10.x.x.x
2015-11-20 08:59:53,377 [SSTable-000] INFO PartitionSelector.java (line 366) allocateBlockBySpace Reallocator(no limit) size=69658934400 initialMarker=1
2015-11-20 08:59:53,377 [SSTable-000] WARN PartitionBlocksContainer.java (line 189) Could not reallocate large blocks due to limitation. Required size 69658934400, large block limit 0, SS 10.x.x.x, partition 1
2015-11-20 08:59:53,378 [SSTable-000] WARN PartitionSelector.java (line 386) Cannot allocate block from partition 1 for size 69658934400 SS 10.x.x.x
2015-11-20 08:59:53,378 [SSTable-000] WARN PartitionSelector.java (line 259) No partition found for reallocating 69658934400, number of partitions 1, SS 10.x.x.x
2015-11-20 08:59:53,378 [SSTable-000] WARN SSTable.java (line 974) Could not find a matching partition using in-memory data for reallocating 69658934400
2015-11-20 08:59:53,378 [SSTable-000] WARN SSTable.java (line 864) No matching block of size 69658934400 found in level 2 DT urn:storageos:OwnershipInfo:9a07987a-c214-40a1-9743-ac8fab113116__SS_29_32_2: for SS 10.x.x.x for reallocation
2015-11-20 08:59:53,378 [SSTable-000] INFO InterSSManagerCommunicationClient.java (line 131) Could not get any blocks though reallocation for SS 10.x.x.x
2015-11-20 08:59:53,379 [SSTable-000] ERROR SSTable.java (line 780) ReallocationTask run() caught unexpected exception:
java.lang.NullPointerException
at com.emc.storageos.data.ssmanager.sstable.SSTable.requestReallocation(SSTable.java:801)
at com.emc.storageos.data.ssmanager.sstable.SSTable$ProactiveReallocator$1.run(SSTable.java:764)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I'm running CentOS 7.1, Docker 1.4.1, and ECS 2.1.0.
The VM has 128GB RAM, 8vCPUs, a 100GB data disk, and almost a whole week of my blood, sweat, and tears.
Has anyone got any suggestions?
Cheers,
Ben
JasonCwik
281 Posts
0
November 20th, 2015 09:00
Ben, can you send me an internal email with your system information? I will have someone take a look.
mumshad
16 Posts
0
April 20th, 2016 13:00
Hi Ben,
Did you figure out how to solve this problem?
Thanks
Mumshad