An older system can be added to a newer Enterprise Manager but not vice-versa.
To add an Avamar 5 system to the Avamar 6 EM, the MCS Web Service SDK needs to be installed on the 5.0 grid. The SDK is available as a hotfix but the 5.0 version can only be installed on top of 5.0.4-30 so this system will have to be upgraded. If the hardware supports 6.0 or 6.1, it might be better to upgrade to one of those releases since they both include the SDK "out of the box". In any case, the hotfix number for 5.0 systems is 26176.