Start a Conversation

Unsolved

This post is more than 5 years old

3383

August 2nd, 2010 06:00

RMAN backups of HP-UX dedicated storage node fail with error "Dedicated storage node leda2 cannot save remote data from client cora127n1.melkweg.tld"

Hello,

I opened a service request (# 35960002) about a problem we're having with RMAN backups on HP-UX cluster nodes. There isn't much progression in the case, so I thought maybe someone here might have some ideas how to solve this puzzle. Problems started when we upgraded the Networker software on the HP-UX nodes from 7.4.3 to 7.5.2. The HP-UX cluster nodes are also dedicated storage nodes. The file system backup is working fine. But the RMAN backups (client initiated) generate this error message:

input datafile fno=00009 name=/d10/oradata/tstcds02/user/ino_index/r152m001

channel c1: starting piece 1 at 02-AUG-10

released channel: c1

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03009: failure of backup command on c1 channel at 08/02/2010 13:27:46

ORA-19506: failed to create sequential file,, parms=""

ORA-27028: skgfqcre: sbtbackup returned error

ORA-19511: Error received from media manager layer, error text:

   Dedicated storage node leda1 cannot save remote data from client cora127n1.melkweg.tld. (0:5:13)

We tried several things. we even re-installed the older Networker version 7.4.3, but still we receive the same error message. The curious thing is that we have a similar HP-UX cluster that we also upgraded along with this cluster and there we have no problem at all.

I hope someone has encountered this problem before and has an answer for this problem.

Regards,

Martijn

PROBLEM DESCRIPTION:
------------------------------------
Fact: NetWorker Storage node for Unix (HP-UX) 7.5.2

Fact: Networker Module for Oracle 5.0

Fact: HP-UX Cluster 11.23

Fact: Dedicated Storage node

Symptom: Error: 'Dedicated storage node leda2 cannot save remote data from client cora127n1.melkweg.tld'

Symptom: Manual backups (RMAN) from clustered dedicated storage node sometimes fail

Symptom: Schedule backups (server initiated) to clustered dedicated storage node succeeds

Symptom: Manual backups (RMAN) from clustered dedicated storage node from a different HP-UX cluster succeed

Change: upgrade Networker software from version 7.4.3 to version 7.5.2

ENVIRONMENT INFORMATION
------------------------------------
< >
Product: NetWorker Storage Node for Unix
Version: 7.5.2

56 Posts

August 2nd, 2010 07:00

Hello Martijn,

As you know Dedicated Storage Node (DSN) is used to store local data ( the client  the storage node is attaches to) . So ensure firstly you have seperate licenses for  DSN.

Secondly ensure that all the dns and hosts files ( name resolution ) is working well.

Also make sure that aliases are set properly, May be there is a mismatch where same box is seen as different one may be because it is multihomed or hostname not translated to FQDN.

26 Posts

August 3rd, 2010 00:00

Hi Bahaa,

Thanks for replying. We have 12 dedicated storage node licenses and 11 dedicated storage nodes. So license-wise I think we're okay. We don't use the hosts file and forward and reverse lookup of the backup server works fine on the HP-UX nodes. The HP-UX storage nodes are in a different DNS zone as the backup server. However we never had any problems before with this and like I said the other HP-UX cluster is in the same DNS zone and there the backup is fine. We might have some sort of breakthrough. I changed 2 things yesterday. I changed the remote access rights on the failing storage nodes. I entered the other storage node to the remote access rights of the first storage node.

storage node1

*@storagenode2

*@storagenode2.local

storage node2

*@storagenode1

*@storagenode1.local

And also I changed storage node1 from a dedicated to a shared storage node. All the backups of storage node2 succeeded and some of the backups of storage node1 failed with error:

ORA-19506: failed to create sequential file,, parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
   Direct saves are not enabled. (0:5:2)

So I changed storage node1 back to a dedicated storage node. I requested to start some RMAN backups of storage node1 and they are running now.

56 Posts

August 3rd, 2010 00:00

Hello Martijn,

Did you try to take filesystem backups on the dedicated storage node ? and what was the results ?

Thanks,

Ahmed Bahaa

26 Posts

August 3rd, 2010 01:00

Hi Ahmed,

The file system backup works fine. We have no problems with the file system backup.

Regards,

Martijn

12 Posts

August 3rd, 2010 01:00

Hi,

can you post client configuration for phisycal hosts and for virtual (cluster) host.

BR

12 Posts

August 3rd, 2010 04:00

Hi,

If you want to use Dedicated Storage Nodes for physical hosts (lan free backup for cluster host) then you have to configure cluster host (virtual)  for Oracle backup as client in Networker and put curphyhost under client Storage node resource.

Then you will have correct licensing, now you are using 2 NMO licenses for one for each physical host, and you need only one if you configure cluster host

The licensing part is true only if you have active-passive cluster.

BR

26 Posts

August 3rd, 2010 04:00

Hi BR,

Here's the client config for both physical nodes of the HP-UX cluster. There's no client config of a virtual (cluster) resource. We only backup the file systems of the physical nodes and the databases are backed up with oracle RMAN.

C:\Windows\system32>nsradmin
NetWorker administration program.
Use the "help" command for help.
nsradmin> show
Will show all attributes
nsradmin> print type: nsr client; name: leda1
                        type: NSR client;
                        name: leda1;
                      server: cora127;
                   client id: \
d60d4028-00000004-48db8a93-48db8e9b-00090000-0a748a7f;
            scheduled backup: Enabled;
                     comment: "CDS:FS:HPUX";
             Save operations: ;
            archive services: Disabled;
                    schedule: Full Every Monday;
               browse policy: 3weeks;
            retention policy: 3weeks;
                  statistics: elapsed = 347581, index size (KB) = 296295,
                              amount used (KB) = 296295, entries = 1847917;
                   directive: Unix standard directives;
                       group: ASD-hpux;
                    save set: /, /var, /usr, /stand, /opt, /home, /d10,
                              /d70/orabck;
  Backup renamed directories: Disabled;
                    priority: 500;
   File inactivity threshold: 0;
File inactivity alert threshold: 0;
               remote access: *@pollux, *@pollux.sfb, *@oberon1,
                              *@oberon1.sfb, *@oberon2, *@oberon2.sfb,
                              *@leda2, *@leda2.sfb;
                 remote user: ;
                    password: ;
              backup command: ;
     application information: ;
                        ndmp: No;
             NDMP array name: ;
       De-duplication backup: No;
         De-duplication node: ;
         Probe resource name: ;
              virtual client: No;
               physical host: ;
           Proxy backup type: ;
           Proxy backup host: ;
             executable path: ;
    server network interface: ;
                     aliases: leda1, leda1.sfb;
                  index path: ;
          owner notification: ;
                 parallelism: 4;
               archive users: ;
               storage nodes: leda1;
       recover storage nodes: ;
         clone storage nodes: ;
                  hard links: Disabled;
             short filenames: Disabled;
                         BMR: Disabled;
                 BMR options: ;
                 backup type: ;
              client OS type: HpOS;
                        CPUs: 2;
           NetWorker version: LNMs_2007.Build.135;
              enabler in use: Yes;
       licensed applications: "NetWorker Module for Oracle, Unix Client/1";
               licensed PSPs: ;

nsradmin> print type: nsr client; name: leda2
                        type: NSR client;
                        name: leda2;
                      server: cora127;
                   client id: \
9bcb14c5-00000004-490977fc-490977fb-05010000-0a748a7f;
            scheduled backup: Enabled;
                     comment: "CDS:FS:HPUX";
             Save operations: ;
            archive services: Disabled;
                    schedule: Full Every Monday;
               browse policy: 3weeks;
            retention policy: 3weeks;
                  statistics: elapsed = 347620, index size (KB) = 223252,
                              amount used (KB) = 223252, entries = 1379407;
                   directive: Unix standard directives;
                       group: ASD-hpux;
                    save set: /, /var, /usr, /stand, /opt, /home, /d10,
                              /d70/orabck;
  Backup renamed directories: Disabled;
                    priority: 500;
   File inactivity threshold: 0;
File inactivity alert threshold: 0;
               remote access: *@pollux, *@pollux.sfb, *@oberon1,
                              *@oberon1.sfb, *@oberon2, *@oberon2.sfb,
                              *@leda1, *@leda1.sfb;
                 remote user: ;
                    password: ;
              backup command: ;
     application information: ;
                        ndmp: No;
             NDMP array name: ;
       De-duplication backup: No;
         De-duplication node: ;
         Probe resource name: ;
              virtual client: No;
               physical host: ;
           Proxy backup type: ;
           Proxy backup host: ;
             executable path: ;
    server network interface: ;
                     aliases: leda2, leda2.sfb;
                  index path: ;
          owner notification: ;
                 parallelism: 4;
               archive users: ;
               storage nodes: leda2;
       recover storage nodes: ;
         clone storage nodes: ;
                  hard links: Disabled;
             short filenames: Disabled;
                         BMR: Disabled;
                 BMR options: ;
                 backup type: ;
              client OS type: HpOS;
                        CPUs: 2;
           NetWorker version: LNMs_2007.Build.135;
              enabler in use: Yes;
       licensed applications: "NetWorker Module for Oracle, Unix Client/1";

26 Posts

August 3rd, 2010 05:00

Hi BR,

The HP-UX cluster is an active/active cluster. I think what you're saying is we use two more licenses than we actually need. My question is does this have any impact on functionality the way things are configured now? Why does one cluster have these problems and another doesn't? What is a curphyhost?

Martijn

12 Posts

August 3rd, 2010 05:00

Hi,

If active/active is used than you need two licenses each for one active node.

But you have to do the backup trough cluster host.

If you don't than index will be written under the physical host, configuration with DSN for LAN free will not work.

Configure new client for each cluster host on NW.

curphyhost is short for current physical host.

In that way (if you put curphyhost)when failover occurs for cluster host from one physical host to the other your LAN free backup will work without intervention.

BR

26 Posts

August 3rd, 2010 06:00

Hi BR,

You are right, this is how we configured all the Windows clusters in Networker as well. However I cannot change the current setup for the HP-UX cluster. What I do find puzzling is that it has worked like this in the past. It is still working on 1 HP-UX cluster, but for some reason it isn't working on the HP-UX cluster concerned anymore. Again thanks for your suggestions.

Martijn

No Events found!

Top