I have not worked on Avamar server 5. If you are not able to find Client Version 6.1 under downloads, then please get in touch with EMC support team. Avamar client version 6.1 is available on EMC support for download but its restricted.
The client catalog on your system would need to be updated before you could use 6.1 clients. Support can assist you with this. That said, unless you have a specific need to remain on 5.0, I would strongly recommend upgrading the system to 6.1 or 7.0.
The 6.1 Client will not work with a 5.x server.
As Ian said, the Client Catalog would have to be updated so the server can accept connections from a 6.1 client.
You can only update the Client catalog by either upgrading the Server to 6.1, or installing the 6.1 Client downloads Only kit.
Both of which require the AvInstaller.
The AvInstaller is only present on Avamar servers running 6.0 or higher.
There is a bootstrap AvInstaller for 5.x systems, but it is only for upgrading the server.
I concur with Ian's recommendation to upgrade the server to at least 6.1.2.