Solved!
Go to SolutionDD prompting for password
My Data domain started prompting for passwords and flat out giving permission denied to a few SQL agent accounts running backup jobs.
The share is r/w to the hosts and I even made the SQL account owner of the share through windows domain admin account.
I can login to the SQL server as the SQL agent account and browse to the share but yet the SQL job still fails.
It's very strange and we have been looking at this for days. It seems to be a problem upstream to the share because tcpdump shows the SMB request never gets past the initial SMB Setup andX Request. Responds with STATUS_LOGON_FAILURE.
account has full read write access. I have made the share completely wide open but it's still getting denied.
2008 Domain Controllers, 208-2012 SQL Server running transact SQL job to dump to the database.
SQL agent is running as an AD service account. Worked for weeks then just stopped at 4AM with no changes.
Tried:
- rebooted DD
- re-joined DD to the domain.
- used cifs troubleshoot to validate the sqlaccount could be looked up from the dd
- used net dns to verify client host and domain controllers resolve.
- Changed ownership of the share to the sql agent.
- updated to ddos 5.2.5 from 5.1
- scratched head vigorously
I went the other way and disabled signing on the client side. That did not work but I will try dd setting. The thing that I don't understand is how the job could start failing in the middle of the night and never work again.
If I setup a new share with the se perms it works. So what is changing that makes the job fail on multiple windows servers ? My instinct is telling me it would be related to a move in as domain controllers to 2008 a few weeks ago and is just now present a problem some how.
Yep, strangest thing.
Same host acls, * for user acl new one works old doesn't.
AND if I delete the old share and create a new one with the same name it still doesn't work.
It's like windows 2008 has remembered something about the share and is using that property incorrectly but I can't see what in tcpdump or in messages with smb logging at 9.
This problem is 2 fold. Enabling signed SMB requires fwd and reverse DNS to resolve to a single interface.
So users who have multiple aliases and possible server issues where a deployment system only updates fwd records signed SMB will stop working. I also had a case where someone re-provisioned a server and it had 2 reverse names so SMB would intermittently fail.
Issue is still ongoing with Signed SMB performance. Looking at disabling it all together.
After weeks of troubleshooting CIFS performance issues I have completely disabled signed cifs support on the data domain and performance has returned to normal. Change was made based on documented performance issues comparing signed SMB1 vs SMB2.
Performance symptoms were evident from SQL backup jobs, and standard UNC copies via the guest OS.
https://www.google.com/#q=signed+smb+performance
Support tested iperf and lmdd tools which proved there were no network or SMB1 limitations but neither tool benchmarked signed SMB.
ScotN
76 Posts
1975
1
Posted June 14th, 2014 12:00
Solution appears to be cifs signing.
Disabling signing on 2008 R2 does NOT work. I disabled this rebooted and it still randomly tried to use a signed SMB connection. Yes, randomly I have SQL agent logs that show hit or miss success with no changes until I set smb signing to auto on the data domain.
#> cifs option set "server signing" auto
Restart cifs.