Start a Conversation

Unsolved

This post is more than 5 years old

1276

November 20th, 2015 01:00

Container restart kills Storage Pool

Hi,

After an epic struggle, I had a working ECS Single Node.  I then (foolishly) restarted the container.  Rookie error.

Since the restart, the GUI is showing the node as "Suspect", and the available capacity of the Pool as 0.0GB.  I need slightly more storage than that to test my application.

Authentication is working via the API, and I can log into the GUI (obviously).  There are a bunch of recurring errors/warnings in the log, but these were showing up prior to the container restart:

2015-11-20 08:59:53,377 [SSTable-000]  WARN  PartitionBlocksContainer.java (line 189) Could not reallocate large blocks due to limitation. Required size 69658934400, large block limit 0, SS 10.x.x.x, partition 1

2015-11-20 08:59:53,377 [SSTable-000]  WARN  PartitionSelector.java (line 386) Cannot allocate block from partition 1 for size 69658934400 SS 10.x.x.x

2015-11-20 08:59:53,377 [SSTable-000]  INFO  PartitionSelector.java (line 366) allocateBlockBySpace Reallocator(no limit) size=69658934400 initialMarker=1

2015-11-20 08:59:53,377 [SSTable-000]  WARN  PartitionBlocksContainer.java (line 189) Could not reallocate large blocks due to limitation. Required size 69658934400, large block limit 0, SS 10.x.x.x, partition 1

2015-11-20 08:59:53,378 [SSTable-000]  WARN  PartitionSelector.java (line 386) Cannot allocate block from partition 1 for size 69658934400 SS 10.x.x.x

2015-11-20 08:59:53,378 [SSTable-000]  WARN  PartitionSelector.java (line 259) No partition found for reallocating 69658934400, number of partitions 1, SS 10.x.x.x

2015-11-20 08:59:53,378 [SSTable-000]  WARN  SSTable.java (line 974) Could not find a matching partition using in-memory data for reallocating 69658934400

2015-11-20 08:59:53,378 [SSTable-000]  WARN  SSTable.java (line 864) No matching block of size 69658934400 found in level 2 DT urn:storageos:OwnershipInfo:9a07987a-c214-40a1-9743-ac8fab113116__SS_29_32_2: for SS 10.x.x.x for reallocation

2015-11-20 08:59:53,378 [SSTable-000]  INFO  InterSSManagerCommunicationClient.java (line 131) Could not get any blocks though reallocation for SS 10.x.x.x

2015-11-20 08:59:53,379 [SSTable-000] ERROR  SSTable.java (line 780) ReallocationTask run() caught unexpected exception:

java.lang.NullPointerException

        at com.emc.storageos.data.ssmanager.sstable.SSTable.requestReallocation(SSTable.java:801)

        at com.emc.storageos.data.ssmanager.sstable.SSTable$ProactiveReallocator$1.run(SSTable.java:764)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)


I'm running CentOS 7.1, Docker 1.4.1, and ECS 2.1.0.

The VM has 128GB RAM, 8vCPUs, a 100GB data disk, and almost a whole week of my blood, sweat, and tears.

Has anyone got any suggestions?

Cheers,

Ben

281 Posts

November 20th, 2015 09:00

Ben, can you send me an internal email with your system information?  I will have someone take a look.

16 Posts

April 20th, 2016 13:00

Hi Ben,

Did you figure out how to solve this problem?

Thanks

Mumshad

No Events found!

Top