Installing OMSA onto a VM will not allow you to manage the hardware. The process to achieve what you're looking for is going to be a bit more involved. There is a package of OMSA that is built for ESXi, you'll need to install that onto the host itself. It does require a reboot, though. Once the package is installed onto the host, then you can install onto the VM. The instance on the VM is going to act as a client, while the instance on the host will act as the server. You'll just need to use the client to select "Manage Remote Node," then navigate to the host.
Looking at our download page, it looks like the last OSMA version 2008 R2 supported was 8.5. If you stick with that version of Windows, I would keep working with OMSA 8.5. The newer versions are more geared towards working with newer iDRACs anyway.
Finally, the login issues. I'm not sure what would be causing the issues you're experiencing. What you should be able to do is login using the credentials of the ESXi host. You might consider getting everything laid out like this, then trying again, so you can see if you get different behavior.