RPC error: RPC program or version mismatch; low version=305, high=305

Do you get this during index save only? I assume, other backups are affected too? (or is it just index)?

Refer to below link, It might help you to fix this issue.

https://support.emc.com/kb/146764

Regards,
Prajith

Hi,

Just adding to discussion here. I use Networker 8.0.1.1 build .132.

Sometimes I get this error during VADP backups. On later restarts it runs fine, I could not figure out why this behaviour.

@Prajith - could you please paste the content of KB article here. I am not able to access it.

Regards

Here is the content,

Program or version mismatch errors can occur for multiple reasons and may be indicative of multiple issues that need to be addressed. In order to provide the most accurate fix, review each of the following resolutions as one or more may apply to the environment:

If you are getting this error only while using NetWorker License Manager (NML) to add/report a license, verify that you are using the appropriate syntax (nsrcap).

1. Restart the NetWorker processes in order to resolve and potential inter-communication issues.

2. Restart the operating system for the machine, if possible.

3. Use hosts file for name resolution between the NetWorker server, client(s), and any Storage Nodes (include reverse name lookup).

4. Verify that for the machine reporting the error message, there are not mixed binary versions. That is, all NetWorker binaries on the client are the same version.

5. Reduce server parallelism as much as possible in order to test if the error is due to resource limitation (network, memory, CPU).

6. Ensure all hops between the client and NetWorker server are set to the appropriate network speed rather than the default of 'Auto Negotiate'.

More Information:

Use rpcinfo to check if there is a problem with the RPC version numbers. The problem could be due to a mismatch in the RPC service requested by client and the service available on the server. Rpcinfo is a command that can be used to check the program and version number of the registered services on a machine.

rpcinfo -p (client_name)

program vers proto port service

100000 3 tcp 36890 portmapper

100000 2 tcp 111 portmapper

100000 2 udp 111 portmapper

390103 2 tcp 846 nsrd

390109 2 tcp 846 nsrstat

390110 1 tcp 846 nsrjb

390103 2 udp 853 nsrd

390109 2 udp 853 nsrstat

390110 1 udp 853 nsrjb

390107 4 tcp 931 nsrmmdbd

390107 5 tcp 931 nsrmmdbd

390104 105 tcp 845 nsrmmd

390104 205 tcp 945 nsrmmd

390105 5 tcp 677 nsrindexd

390113 1 tcp 7937 nsrexec

"nsrd", always has "390103" for program number and "2" for version number. 

The "recover" command from client, in turn, uses rpcinfo to get service from the NetWorker server. If the program and version numbers do not match, the error message, "RPC: Program/version mismatch;", can be returned.

That is, the following shows how the error message is returned when the machine "goldfarb" is "pinged" for "nsrd".

>rpcinfo -t goldfarb 390103 2 program 390103 version 2 ready and waiting

If we request "nsrd" with version "3" (and not "2"), the following is the error obtained:

>rpcinfo -t goldfarb 390103 3

rpcinfo: RPC: Program/version mismatch; low version = 2, high version =2 program 390103 version 3 is not available

Hello guy's

Prajith your recomendation apllies also to this?

RMAN-03009: failure of backup command on IncrEIDC4 channel at 01/20/2014 20:08:07

ORA-19506: failed to create sequential file, name="DifCumul_EIDC1_7gouikt3_1_1", parms=""

ORA-27028: skgfqcre: sbtbackup returned error

ORA-19511: Error received from media manager layer, error text:

  RPC program or version mismatch; low version = 123605, high version = 123605 (0:5:2)

channel IncrEIDC4 disabled, job failed on it will be run on another channel

channel IncrEIDC1: finished piece 1 at 20-JAN-14

piece handle=DifCumul_EIDC1_7douikt3_1_1 tag=TAG20140120T200747 comment=API Version 2.0,MMS Version 1.1.0.0

channel IncrEIDC1: backup set complete, elapsed time: 00:00:29

channel IncrEIDC1: starting incremental level 1 datafile backup set

channel IncrEIDC1: specifying datafile(s) in backup set

input datafile file number=00002 name=/u02/oradata/EIDC1/sysaux01.dbf

channel IncrEIDC1: starting piece 1 at 20-JAN-14

RMAN-03009: failure of backup command on IncrEIDC3 channel at 01/20/2014 20:09:27

ORA-19506: failed to create sequential file, name="DifCumul_EIDC1_7fouikt3_1_1", parms=""

ORA-27028: skgfqcre: sbtbackup returned error

ORA-19511: Error received from media manager layer, error text:

  RPC program or version mismatch; low version = 124605, high version = 124605 (0:5:2)

channel IncrEIDC3 disabled, job failed on it will be run on another channel

RMAN-03009: failure of backup command on IncrEIDC2 channel at 01/20/2014 20:11:03

ORA-19506: failed to create sequential file, name="DifCumul_EIDC1_7eouikt3_1_1", parms=""

ORA-27028: skgfqcre: sbtbackup returned error

ORA-19511: Error received from media manager layer, error text:

   Direct saves are not enabled. (0:5:2)

Best regards

Hi Fernandes,

     Is this a new configuration or has this worked before, can you check if client direct is enabled ?

Good morning,

I have nw 7.6.5 this was work now fail with this error...

Melhores cumprimentos/ Best regards

Ricardo Fernandes -EMC

#This email was send by a mobile device sorry for any typos.

You might well try what Prajith has recommended.

We just started to see this same error on one of our Networker servers as well. Seeing it on multiple clients but not all. Some Windows clients and some Oracle RMAN backups.

Just curious if anyone has tried restarting Networker on the Networker server or rebooting the Networker server?

Please provide environment information.

Networker server running 7.6.1

on Linux

# uname -mrs

Linux 2.6.9-89.0.20.ELsmp x86_64

In some cases we see the error only on the index save and all other save sets are good..

Here we saw it on vss save sets:

98520:save: Falling back to agent-assisted indirect save

90095:save: Cannot open save session with 'qdcls0228.us.qdx.com': RPC program or version mismatch; low version = 10305, high version = 10305

76677:ansrd: RPC program or version mismatch; low version = 10305, high version = 10305

Here we saw it on an ndmp backup...

5767:nsrdsa_save: NSR server qdcls0228.us.qdx.com: busy

5766:nsrdsa_save: waiting 53 seconds then retrying

39078:nsrdsa_save: RPC error: RPC program or version mismatch; low version = 10405, high version = 10405

5777:nsrdsa_save: Cannot open nsrdsa_save session with qdcls0228.us.qdx.com

Here we saw it on an RMAN backup....

ORA-19506: failed to create sequential file, name="/FULL_APEXP_2rps4lik/", parms=""

ORA-27028: skgfqcre: sbtbackup returned error

ORA-19511: Error received from media manager layer, error text:

   RPC program or version mismatch (0:5:2)

Usually that shows mismatch in cached information and the one being actual in RPC table.  Quick win is to restart NW as following:

- stop NW server

- Stop SN

- remove content of /nsr/tmp on each of these

- make sure rpcinfo -p localhost on each of these does not show any 390xxx process

- start SN

- start NW server

NW 7.6.1 is rather old and I would not running on kernels 2.6.x - would rather go for NW8 which is supported only now.  In 8.2 I believe EMC no longer uses rpcbind, but rather their own rpc control. 

Thanks, Hrvoje for the reply.  I will give this a shot and let you know the results.

Yes, this is a very old system/environment that needs a refresh. Hopefully this year...

Hi Wendler,

This is a documented issue, run this command please:

nsrinfo -n oracle CLIENTNAME

the outuput should something like this "can't read the index client"

so to solve it run this command:

nsrck -L 5 CLIENTNAME

Please see if solve your problem and say the result.

Cheers,

Hi Rtfernandes,

The nsrinfo -n oracle CLIENTNAME command was successful..

[root@qdcls0228 ~]# nsrinfo -n oracle apexdb

scanning client `apexdb' for all savetimes from the oracle namespace

c-3524577632-20150107-00, date=1420685888 Wed 07 Jan 2015 09:58:08 PM EST

s5ps5djf_1_2, date=1420685883 Wed 07 Jan 2015 09:58:03 PM EST

s3ps5d8k_1_2, date=1420685778 Wed 07 Jan 2015 09:56:18 PM EST

s4ps5d8k_1_2, date=1420685723 Wed 07 Jan 2015 09:55:23 PM EST

s2ps5d8h_1_2, date=1420685719 Wed 07 Jan 2015 09:55:19 PM EST

s1ps5d7d_1_2, date=1420685704 Wed 07 Jan 2015 09:55:04 PM EST

s0ps5d7d_1_2, date=1420685590 Wed 07 Jan 2015 09:53:10 PM EST

c-3524577632-20150106-00, date=1420599498 Tue 06 Jan 2015 09:58:18 PM EST

rups2p6q_1_2, date=1420599464 Tue 06 Jan 2015 09:57:44 PM EST

etc....