Celerra Simulator Linux kernel not SMP

Hi Marc,

Sorry, the kernel is specialized for the simulator and source code is not available.

The simulator should not be expected to perform as fast as a real Celerra since it not only runs the Control Station functions but also the Dart(s). It also is running on top of a virtual environment - though the performance in ESX should be better than Vmware Workstation. Can you tell me specific functions that are slow?

Thanks,

Jim

Hi Jim,

Sorry, I thought that the sources were in /usr/src/kernels/2.6.9-22.0.blackbird-i686 ...

When I say that the simulator is slow I talk about Java interface.

It is really strange: sometimes it works very well sometimes it is very slow (cpu spikes to 100% in virtual center performance monitor)
I cannot determine if there is a particular function causing these slowdowns.
I will investigate and I come back if I find a cause.

Thanks
Marc

Hi Marc,

I thought you had me for a minute when I found some .c files under that dir. but those are not part of the kernel! .

Java stuff is sometimes slow. I don't play in that area so I don't know what would cause your performance problems. Since you're running on ESX, I don't think this is your problem, but when it first boots up, there is a lot of stuff going on and you have to wait several minutes for things to settle out.

That's really all I can say now. If I find out any more, I'll post it here.

Jim

Hi Jim,

I made a screenshot of the CPU performance :
http://img53.imageshack.us/img53/2208/celerrajr4.jpg

When I start the VM, the CPU is 100% during 15 minutes (1 Core Xeon E5130 = 1600MHz )
Then VM is 25% CPU while I do nothing (which seems a lot?)
Once I start the Celerra manager in my browser, the VM back to 100% and the manager is slow but I do nothing special, just browse left menus.

Do you have the same behaviour ?

Is this due to the fact that I can use a single core not very powerful?
And since the kernel of the simulator is not SMP I am condemned to remain like this or there's a solution?

Thanks
Marc

Hi Marc,

First, I wasn't able to view the jpg you posted because our internal security system was afraid that it was spyware! I'll try to look at it when I get home this evening or this weekend. I assume that it's a screen shot from your host PC's task manager or equivalent and so is a reading from the real CPU and not the Simulator's CPU.

That being the case, it should be kept in mind that the Celerra Simulator is not fast. After all it is running in a virtual machine that is running Linux on a real PC that is running some version of Windows.The obvious thing to do (other than run it on a faster box) is to run the "thin" version. This has only one Data Mover rather than two and is a lot faster. You could also run the Celerra Manager on another PC than the one that is running the Simulator. Make sure that only essential applications are run on the host PC.

Does that help?

Jim

Hi Marc,

I forgot to ask how much memory you have installed in your host PC. If you less than 2GB, both the virtual machine and your host PC could be swapping (which equates to trashing) and that will eat performance better than anything. My suggestion to get a single (thin) Data Mover version would help here.The two Data Mover version virtualizes 2GB while the one Data Mover version only virtualizes 1GB so your memory footprint is much smaller.

I hope that helps,

Jim

Hi Jim,

The simulator that I use is provided by Chad at this address:
ftp://ftp.documentum.com/vmwarechampion/Virtual_Machine_Library/Celerra/Celerra_56_VSA.zip

As advised, I have allocated 3GB of RAM

About your answers:
- I did tests on an empty ESX 3.5 Update 3, only the simulator run on it, so no problem of overload.
- I checked by doing a "top-d 1", the VM does not swap


I tested various versions of the simulator (Chad, OneDM Workstation) on different platforms (ESX, VMServer) on different CPU (Xeon,P4,Xeon quadcore) and it is still slow:

An example:
in the video of Chad "Step 4: making your Celerra single VM." nas_cel -update id = 0 takes 29 seconds
when I run this command, it takes more than 2 minutes.

Conclusion:
I guess simulator need a lot of power. With only one core usable, my QuadCore Xeon CPU has not enough power to run the sim.
As long as the kernel of the simulator will not be SMP, for me it will be barely usable.

Thanks for your patience, your answers (and sorry for my bad english)
Marc


PS: Two little points:

1 / On Chad's version (Celerra_56_VSA.zip), I notice 2 blackbird process including one with server_3 while this version is OneDM?

On Workstation version that I downloaded on PowerLink (Celerra_Simulator_OneDM.zip), this 2nd process does not exist.

----- Chad version ------------------------------------
[root@smrfpfinns01 ~]# ps aux |grep black
root 2845 0.1 0.0 3404 900 ? S Dec08 1:32 ./blackbird_control -s -m
root 3210 0.5 20.1 1534552 627504 ? S Dec08 6:25 ./slot_3/blackbird_serv_3 -i 3 -h 25 -w 80 -s /dev/pts/20 -C3 -f slot_3/boot.cfg -L -S600?
root
27869 1.4 20.2 1534552 630904 ? S 13:42 0:28 ./slot_2/blackbird_serv_2 -i 2 -h 25 -w 80 -s /dev/pts/20 -C3 -f slot_2/boot.cfg -L -S600?

[root@smrfpfinns01 bin]# ./server_version server_2
server_2 : Product: EMC Celerra File Server Version: T5.6.36.2
[root@smrfpfinns01 bin]# ./server_version server_3
server_3 :
Error 4023: server_3 : unknown host

--------Workstation Version----------------------------------------------
[root@localhost ~]# ps aux |grep black
root 2935 0.2 0.0 3256 832 ? S 08:11 0:06 ./blackbird_control -s -m
root 3027 6.5 24.5 1534552 254628 ? S 08:11 3:17 ./slot_2/blackbird_serv_2 -i 2 -h 25 -w 80 -s /dev/pts/20 -C3 -f slot_2/boot.cfg -L -S600?


2 / I noticed every second, there is a new ping process

I don't know what it is and if it has influence or not on the VM

------------------------------------------------------
[root@smrfpfinns01 bin]# while true; do ps aux |grep ping |grep -v grep; sleep 1; echo ----------- ;done
root 9697 4.0 0.0 0 0 ? Z 15:04 0:00 [ping]
-----------
root 9705 5.0 0.0 0 0 ? Z 15:04 0:00 [ping]
-----------
root 9722 0.0 0.0 0 0 ? Z 15:04 0:00 [ping]
-----------
root 9729 0.0 0.0 0 0 ? Z 15:04 0:00 [ping]
-----------
-----------
root 9742 0.0 0.0 0 0 ? Z 15:05 0:00 [ping]
-----------
root 9750 0.0 0.0 0 0 ? Z 15:05 0:00 [ping]
-----------

Hi Marc,

While Simulator performance has never been expected to be fast (in fact we do no performance testing on it, though we are looking into some options in that area for the future), it should be usable with a reasonable CPU. I'm at a bit of a disadvantage discussing actual CPUs because I don't know how to compare what I have a 2.3 GHz 6550 and it works quite well, but I've used much less powerful boxes with "reasonable" results. Usually, the biggest problem is with the amount of memory (all other things equal) simulated in Vmware and installed on the host PC. I would not monkey with the amount of memory simulated by Vmware and the more memory on the host, obviously the better but for a 2 DM Simulator, you should have at least 3 GB installed on your PC. All this supposes you are running Vmware Workstation or Player on a PC. I have very little experinece with running the Simulator on ESX but it should be faster, not slower than with Workstation or Player.

The Simulator you got from Chad is a 2 DM Simulator. Those two blackbird_serv processes are the Darts. You can also see the same thing by running the /nas/sbin/getreason command. That is certainly part of your problem. I don't know what changes he's made to the ones he distributes (maybe he can chime in here), but I would advise using the *oneDM* version from Powerlink if that's possible.

I hope this helps and you can get acceptable performance on your Simulator.

Good luck,

Jim

Jim,

I followed your advice and I tested "oneDM" version from Powerlink : same results as Chad version : slow and a second blackbird process....

I understood why there's a 2nd blackbird process

Have a look at /etc/init.d/blackbird
------------------------------------------------------------------------
[...]
./blackbird start serial 2 > /dev/null 2>&1
ret=$?
free_mem=`/bin/cat /proc/meminfo |grep MemTotal | awk '{print $2}'`
if [ $free_mem -gt 2000000 -a $ret -eq 0 ]; then
sleep 5
./blackbird start serial 3 > /dev/null 2>&1
ret=$?
if [ $ret -eq 0 ]; then
DARTS=2
fi
fi
[...]
------------------------------------------------------------------------
With OneDM or Chad version, if you have more than 2Gb of memory allocated, the simulator start automatically a 2nd DM

I agree that simulator performance are slower than real Celerra. In fact, I have not yet tested CIFS / NFS / iSCSI features, I just try to have a java interface faster than 1 minute between each click

I commented the lines above in /etc/init.d/blackbird and java gui seems to be faster.

Marc

Hi Marc,

I'm very familiar with that code (I wrote some of it ). The 1DM Simulators should simulate less than 2GB of memory - 1GB if I recall. While you can safely go from a 1DM Simulator to a 2DM Simulator by increasing the amount of simulated memory and rebooting, it is not advisable to go the other way. Once the Simulator has 2 DMs, it always thinks it has 2 even though you may have reduced the amount of memory so that only one DM boots. That is the effect of what you did when you commented out that code. As the code shows, you can find out how much memory your Simulator is simulating by looking at the Memtotal field of the file, /proc/meminfo. If you downloaded the 1DM Simulator from Powerlink and you're getting two blackbird_serv* processes, there could be a problem with the build process. I'll check it out.

Jim

Message was edited by:
James Grunewald

Message was edited by:
James Grunewald

Marc,

Did you get your 1DM Simulator from here:

http://powerlink.emc.com/km/live1/en_US/Offering_Technical/Software_Download/Celerra_Simulator_OneDM.zip?mtcs=ZXZlbnRUeXBlPUttQ2xpY2tTZWFyY2hSZXN1bHRzRXZlbnQsZG9jdW1lbnRJZD0wOTAxNDA2NjgwMzNhYjk1LGRhdGFTb3VyY2U9RENUTV9lbl9VU18w

This one worked for me.

Jim

Hi Jim,

First, thank you for helping me

I get the last version of OneDM available on Powerlink:
md5sum : cc588003b3b482f4e770a5eb32cfd864 Celerra_Simulator_OneDM.zip

I realized my mistake: The first thing I did before starting the VM is to edit settings and change the default amount of memory from default (1024) to 3072. That's why I had a second blackbird process eating CPU and memory.

Why I did that ? My first goal is to test replicator, so I followed this advice :

"Either way - 2GB of RAM is a minimum - and I would strongly recommend 3GB. 3GB is a minimum if you're going to use Celerra Replicator (i.e. replicate between nodes). EMC has a legacy as a hardware company, and real Celerras all ship - even the itty bitty ones - with 5 CPUs or more, ten's of GB of memory, etc).
[...]
For masochists out there, there is a "dual datamover" VMware Workstation format (think HA) version of the hardware version of the simulator for developers writing code and who want to test.
"

I have two questions:
1/ Using OneDM version with more than 2Gb of memory, why do you start automatically a second "sleeping" DM ? If I wanted a second DM, I took the TwoDM version ?
I'm sure there's a good reason but it's confusing me.

2/ Corrects me if I am wrong because I do not know quite well EMC products, that's why I tested simulator but :
- To test replicator I just need 2 "OneDM" VM ? I don't need 2 "TwoDM" VM ?
- If i'm right, I will use OneDM version and I shouldn't follow the above advice to change memory because it will start a 2nd unnecessary DM?

Thanks,

Marc

Hi Marc,

I really screwed up the formatting on this thread with that link

To your first question, in 2 DM Simulators, the second one isn't sleeping, it's the secondary by default (that can be changed so they both are active servers). The amount of simulated memory is a convenient way of automatically regulating how many DMs to boot. I'm not experienced in replication (other than I know that it works), but I think the real parameter is the amount of memory the Dart (DM) uses. One person I talked to said he had to bump it up to 1.2GB for replication. This is separate from the memory we've been talking about (PC memory and virtual machine memory). You can change the amount of memory the Dart uses by editing the /opt/blackbird/blackbird_setup file. Change the value for BB_MEMSIZE to the desired size in MB. Then bounce your Dart (while still in the /opt/blackbird directory) running:

./stop 2
./start 2

Obviously, you should pay attention to how much memory your virtual machine is simulating so that you don't make the Dart so big that you swap a lot.

Another thing to think about is that you can just stop one of the Darts in a 2DM Simulator by going to /opt/blackbird and running stop 3 (to stop server_3). This should put less strain on your CPU and since your dual DM Simulator is set up for more simulated memory, you have a bit more leeway in sizing your remaining Dart. I didn't think of this because I almost always use a 1DM Simulator.

By the way, the Datamovers on real Celerras have either 4 GB or if it's a newer one, it has 8 GB. Not tens of GB.

Does this help?

Jim

Hi Jim,

Thanks to you, now I better understand the management of DMs and memory.

To summarize:
I want to keep only one DM for performance reasons so I'll configure a little less than 2GB of RAM in VM (to prevent automatic booting of 2nd DM), I will edit value BB_MEMSIZE in /opt/blackbird/blackbird_setup from 600MB (default) to 1200Mo (for replication) and I'll check that VM doesn't swap a lot.

The advice above that said to use 3GB of RAM in the VM and that real Celerras have tens of GB come from Chad's blog :
http://virtualgeek.typepad.com/virtual_geek/2008/06/get-yer-celerra.html
Maybe he should provide some details

Thanks for all !
Marc

Hi Marc,

That sounds good, but I think it would be better to start with a 1DM version and increase the simulated memory and the Dart size. This will keep the Simulator from thinking it is supposed to have two DMs. I don't know if it thinking there's supposed to be two DMs will be a problem most of the time, though.

Let me know how things work out.

Jim